AI LLM via Canister/Cycles?

[Gekctek]

I see a lot of AI projects out there working on LLM solutions built on the internet computer. I was wondering if anyone was aware of any project that is working on one that allows for payment of generation with cycles.
I’m thinking the context of a DAO controlled application where I do not want to create an account or have API keys or billing outside of the application but rather pay in cycles.

[skilesare]

I have some ideas for a set of icrc standards for describing, providing and tracking this kind of payment. Would be happy to put our heads together on it.

[Gekctek]

I was thinking of just sending cycles for the request that would pay for the computation/offset costs directly, but I guess since things wont be on-chain like GPUs, then it wouldnt work natively, unless they take cycles that would be an equivalent and do it indirectly.

So are you suggesting more of a payment model where the DAO could send ICP or other crypto to pay for certain amount/subscription? Is there anything right now that does something like this (not just AI)?

[skilesare]

I was more thinking discovery and registration/proof of cycles. Cycles ledger is going to be a thing. How many cycles do I send to call do_a_thing()? Is it per byte sent? Can I pre send cycles so I’m authorized? If I’m a canister can I include them with the call? Basically a menu and methods of payment.

[Gekctek]

Sounds cool. Id be down to help

[Icdev2dev]

That would be cool.

I think that the main challenge would be : how to charge; given that the token count is variable out of the LLM and the call charges are based on the token count. Of course the call can be interrupted if one reaches a limit.

Additionally the call should be made with a “transaction id” from IC so that the recieving application makes the call once per transaction for the first replica and caches the result for all subsequent replicas.

[icarus]

This has been an ongoing discussion topic in the DeAI working group this year about a mechanism to pay in cycles for query calls above the call instruction limit, for running LLMs in canister.
The forum conversation starts here after our early Jan meeting:

Technical Working Group DeAI

Thank you for finding the Query Charging forum thread and cross-posting. That thread provided insightful information on the topic: Current Situation: As we know, canisters on the IC are only charged for updates, not for queries. Dfinity agrees that this leads to an imbalance as canisters with more update traffic bear higher costs, inadvertently subsidizing those with more query traffic. This discrepancy, which originated from technical challenges at IC’s launch, contradicts the principle of fa…

This then to further discussion in the Explore Query Charging thread here:

Community Consideration: Explore Query Charging

Introducing an Opt-In Mechanism for Query Charging The discussion in this thread has revolved around whether to enable query charging for all canisters or for none. In contrast, I propose an opt-in mechanism. By integrating a system where canisters can opt-in to pay for queries, we can tailor the IC’s capabilities to diverse needs and use cases, especially in the realm of advanced AI applications. Introducing options like ic_cdk::query(cycles_fee=true) or ic_cdk::query(cycles_fee=true, composit…

The Omnia Network team ( @ilbert and @massimoalbarello ) have been working on a project to enable paying (with ICP) for compute & web services external to the IC, based on their WebSocket Gateway work and following on from this proposal thread they started:

Proposal: Enable canisters to pay in ICP for external services

Objective Enable canisters to pay in ICP for computing services outside the Internet Computer. Background After participating in the ICP.Lab Gaming & Metaverse, @ilbert and I realized that many developers in the IC ecosystem still rely on AWS - or similar services - for some parts of their dapps. This is due to the IC being not yet mature enough to serve all their needs. Current services outside of the IC include the HTTP Gateway and WebSocket Gateway - and any other networking protocol gateway…

I think you are across some or most of this but may be useful for others coming reading this thread.