AI LLM via Canister/Cycles?

[icarus]

This has been an ongoing discussion topic in the DeAI working group this year about a mechanism to pay in cycles for query calls above the call instruction limit, for running LLMs in canister.
The forum conversation starts here after our early Jan meeting:

Technical Working Group DeAI

Thank you for finding the Query Charging forum thread and cross-posting. That thread provided insightful information on the topic: Current Situation: As we know, canisters on the IC are only charged for updates, not for queries. Dfinity agrees that this leads to an imbalance as canisters with more update traffic bear higher costs, inadvertently subsidizing those with more query traffic. This discrepancy, which originated from technical challenges at IC’s launch, contradicts the principle of fa…

This then to further discussion in the Explore Query Charging thread here:

Community Consideration: Explore Query Charging

Introducing an Opt-In Mechanism for Query Charging The discussion in this thread has revolved around whether to enable query charging for all canisters or for none. In contrast, I propose an opt-in mechanism. By integrating a system where canisters can opt-in to pay for queries, we can tailor the IC’s capabilities to diverse needs and use cases, especially in the realm of advanced AI applications. Introducing options like ic_cdk::query(cycles_fee=true) or ic_cdk::query(cycles_fee=true, composit…

The Omnia Network team ( @ilbert and @massimoalbarello ) have been working on a project to enable paying (with ICP) for compute & web services external to the IC, based on their WebSocket Gateway work and following on from this proposal thread they started:

Proposal: Enable canisters to pay in ICP for external services

Objective Enable canisters to pay in ICP for computing services outside the Internet Computer. Background After participating in the ICP.Lab Gaming & Metaverse, @ilbert and I realized that many developers in the IC ecosystem still rely on AWS - or similar services - for some parts of their dapps. This is due to the IC being not yet mature enough to serve all their needs. Current services outside of the IC include the HTTP Gateway and WebSocket Gateway - and any other networking protocol gateway…

I think you are across some or most of this but may be useful for others coming reading this thread.