Introducing the LLM Canister: Deploy AI agents with a few lines of code

Developers
,
23

I noticed that as well. It happens a non-negligible number of times. I think this is a problem with the implementation of the sampler and not with the model itself. Looking into it.

Apart from the times where it returns gibberish, how did you find the model itself?

2 Likes
24

I think it works quite well for generating tweets based on a persona. Soon, we will start using them to analyze prediction markets, which will be a bit more challenging since they will need to estimate probabilities based on available data and compete.

1 Like
25

If a closed model is not possible, it would be nice to have DeepSeek working.

1 Like
26

@Mar Done some changes, and I think now you shouldn’t be seeing gibberish anymore, or at least it should be way less probable now. We’ll have a more permanent solution to that problem very soon.

@marcio I’ve tried DeepSeek extensively, and it’s impressive. We’ll explore supporting one of their distilled versions.

3 Likes
27

Thanks for building this @ielashi!
We’ve just launched the “LlamaBot” within OpenChat which allows users to easily send prompts to the LLM canister!

Screenshot 2025-02-28 at 23.40.43
Screenshot 2025-02-28 at 23.40.43958×404 36.6 KB

7 Likes
28

How were you able to build a bot for openchat with it @ielashi

29

I built it, you can find the source code here - open-chat-bots/rs/offchain/examples/llama at main · open-chat-labs/open-chat-bots · GitHub

6 Likes
30

Quick update: we released a new example of an agent, just to showcase what it’s like to build an agent that specializes in a specific task. In this case, the task is to lookup ICP prices.

A Rust and a Motoko implementation are provided in the examples folder here.

screenshot
screenshot1554×1796 284 KB

4 Likes
31

Indeed, we need to use better models for this, as the difference in probability estimates is too large (compared to 4o, for instance).

32

Hey everyone,

I’d like to get some feedback here on what you think should be the highest priority for the LLM canister. There will be continuous work on all the topics below, but I’d like to gauge which one is the most important to you at this stage:

  • Support for more models (still centralized, managed by DFINITY)
  • Decentralizing the AI workers (models will no longer be managed centrally by DFINITY)
  • API improvements (e.g. support for tools, more tokens per request, images, etc.)
0 voters

Would love to hear also any comments you have wrt your choice.

1 Like
33

I think supporting more models is the best choice in the near term. Then, we can spend more time decentralizing, plus we gain more time until open-source models catch up.

34

Thanks everyone for voting, I appreciate your input. I closed the poll now.

As I mentioned, we’ll be making progress across all three fronts, but this poll will definitely help with the prioritization.

Regarding bigger models: can you share which models you’d like to have access to? To allow for decentralization later on, we can only consider open weight models.

Regarding decentralizing the AI workers: I’ll start a separate thread about that, as it’s a bigger topic.

Regarding API improvements: We just released a typescript library to seamlessly integrate with Azle, so developers don’t need to learn Motoko or Rust to build AI agents on the IC. Checkout the quickstart example that can be used as a template for building agents in Typescript (cc @tiago89).

Prompting (single message)

import { IDL, update } from "azle";
import * as llm from "@dfinity/llm";

export default class {
  @update([IDL.Text], IDL.Text)
  async prompt(prompt: string): Promise<string> {
    return await llm.prompt(llm.Model.Llama3_1_8B, prompt);
  }
}

Chatting (multiple messages)

import { IDL, update } from "azle";
import { chat_message as ChatMessageIDL } from "azle/canisters/llm/idl";
import * as llm from "@dfinity/llm";

export default class {
  @update([IDL.Vec(ChatMessageIDL)], IDL.Text)
  async chat(messages: llm.ChatMessage[]): Promise<string> {
    return await llm.chat(llm.Model.Llama3_1_8B, messages);
  }
}
2 Likes