Introducing the LLM Canister: Deploy AI agents with a few lines of code

Developers
,
63

Hey @dawnnguyen, sorry for the late reply, I was on PTO. You’ve precisely described the approach and how it’s implemented on the DFINITY side.

The main limitations around input/output limits, context length, and model diversity stem from cost, maintenance, and latency considerations. If cost and maintenance aren’t concerns for you, you’re free to choose whatever serves your purpose best.

One key limitation is that data streaming isn’t supported. Unlike the experience you get when querying ChatGPT, where the response appears character by character as it’s generated, you’ll need to wait for the complete output text. This can result in noticeable delays depending on your model size, complexity, and input length.

Without going into too much detail, additional considerations might be:

  • Reliability: The off chain worker becomes a single point of failure. Make sure to monitor, have error handling, or redundancy if reliability is important.
  • Scalability: If usage grows, you might need to redesign the off-chain worker to scale accordingly.
2 Likes
64

@aespieux The issue with dfx deps pull failing on versions 0.26.1 and higher has been resolved in dfx 0.28.0. The problem was related to the switch from replica to PocketIC in versions above 0.25.0, which caused topology differences that prevented dependency pulling.

You should now be able to upgrade to dfx 0.28.0 and successfully pull the llm canister dependency. Please let me know if you encounter any issues after upgrading.

2 Likes
65

Hi @ddave, after implementing a POC, I encounter some issue, could you please give me some advice?

  1. I have a state type that mark the message is queued, processing or responded. And at step Off-chain worker poll the queued message from the LLM canister, I update the state to processing, so that it has to be an update, not a query call => if offchain service continuous polling, it might increase the price due to update call. Are you having the same problem? I’m thinking about keeping the poll function query call, and the offchain worker will filter out the processing msg base on its id, but not sure will it work. Can you share how you handle this?
  2. How can the LLM canister call the chatbot canister to update the responded message? Currently my offchain service can call the LLM canister to poll and update message from LLM canister, but I can’t update the main canister from LLM canister
66

Hey @dawnnguyen

  1. You can have a query call that does the polling. When it hits a message it wants to process, you can then do an update call to update the state.

  2. The example we provide requires you to have a blocking call from yourDapp → LLM canister that will wait until the off-chain worker picks up the message (query/update) and responds with the update call to put it back into the LLM. That’s why there is a timeout that will wait to process the prompt sent to the LLM canister. If that flow doesn’t work for you can add an endpoint to your dapp that will allow it to put responses into it (which the LLM canister can do with an intra-canister call).

Let me konw if that makes sense.

67

@ddave thanks for your answer. The 2nd is very interesting, I will try both solution (blocking until get the response and intra-canister call) and will update the the result later. Personally, I prefer the intra-canister call than the blocking method, but I will measure more to choice the best way to safe cycle

1 Like