Those of you who are into conversational AI are probably aware of this model that has taken the open source world by storm: karpathy/llama2.c Using icpp-pro , I am able run this LLM in a canister and do inference: dfx canister call llama2 inference '(record {"prompt" = "" : text; "steps" = 20 : nat…

@ildefons , Right now, the inference call is a query call, so it does not need to go through consensus. It is very fast to get the response back from this tiny LLM. I plan to activate the timer, and I can give you the tokens/sec. Good question though about what would happen if we switch to an upd…

Very nice! Can’t wait to see more your progress :+1: :+1:

This is VERY cool. Can we chat about it? You can DM me @BobBodily on Twitter, Discord, or Telegram. Would love to learn more.

Try it Deployment to main-net went smooth. icpp_llama2 with the stories15M.bin is now running on-chain in canister 4c4bn-daaaa-aaaag-abvcq-cai. You can call it’s inference endpoint with: dfx canister call --network ic 4c4bn-daaaa-aaaag-abvcq-cai inference '(record {prompt = "" : text; steps = 20 …

I like to use this thread to share the high level roadmap I have in mind for icpp-llm, and keep you posted on the progress each time I reach a milestone or encounter a blocker. Milestone 1: remove the limitations of the current tinystories canister, so it can generate stories longer than 20 words. …

COMPLETED Milestone 1: remove the limitations of the current tinystories canister, so it can generate stories longer than 20 words. I fixed it by: Switching from using a single query call to a sequence of update calls Save the LLM’s runState in orthogonal persistence Implement a new endpoint new_…

Thanks for sharing the progress! Looking forward to trying on mainnet!

Multi-user is now supported by icpp-llm and it is again deployed to main-net canister 4c4bn-daaaa-aaaag-abvcq-cai ! It is deployed to main-net, and to see it in action, check out this short loom video , where I run two concurrent shell scripts, and two stories are being generated by the LLM for two…

Hey @icpp , how do you handle randomness at the moment? Is the same prompt giving the same answer or is icpp taking care of this and uses the randomness API of the Internet Computer?

Llama2.c LLM running in a canister!

Developers Programs & Applications

ildefons August 7, 2023, 9:18pm 2

What is the average inference time? If it takes longer than the consensus period, what happens?

Topic		Replies	Views
Llama.cpp on the Internet Computer Programs & Applications	19	1142	July 28, 2026
Introducing the LLM Canister: Deploy AI agents with a few lines of code Developers rust , DeAI	76	5332	September 1, 2025
AI and machine learning on the IC? Developers	114	10975	June 20, 2024
How can I take an open source pretrained LLM model, deploy it to ICP and use as a private ChatGPT just fo me Developers	13	577	June 16, 2025
Is llama-13B(or 7B) LLM possible to deploy on canister? Developers Discussing	5	586	October 8, 2025

Llama2.c LLM running in a canister!

Related topics