Those of you who are into conversational AI are probably aware of this model that has taken the open source world by storm: karpathy/llama2.c Using icpp-pro , I am able run this LLM in a canister and do inference: dfx canister call llama2 inference '(record {"prompt" = "" : text; "steps" = 20 : nat…

@ildefons , Right now, the inference call is a query call, so it does not need to go through consensus. It is very fast to get the response back from this tiny LLM. I plan to activate the timer, and I can give you the tokens/sec. Good question though about what would happen if we switch to an upd…

Very nice! Can’t wait to see more your progress :+1: :+1:

This is VERY cool. Can we chat about it? You can DM me @BobBodily on Twitter, Discord, or Telegram. Would love to learn more.

Try it Deployment to main-net went smooth. icpp_llama2 with the stories15M.bin is now running on-chain in canister 4c4bn-daaaa-aaaag-abvcq-cai. You can call it’s inference endpoint with: dfx canister call --network ic 4c4bn-daaaa-aaaag-abvcq-cai inference '(record {prompt = "" : text; steps = 20 …

I like to use this thread to share the high level roadmap I have in mind for icpp-llm, and keep you posted on the progress each time I reach a milestone or encounter a blocker. Milestone 1: remove the limitations of the current tinystories canister, so it can generate stories longer than 20 words. …

COMPLETED Milestone 1: remove the limitations of the current tinystories canister, so it can generate stories longer than 20 words. I fixed it by: Switching from using a single query call to a sequence of update calls Save the LLM’s runState in orthogonal persistence Implement a new endpoint new_…

Thanks for sharing the progress! Looking forward to trying on mainnet!

Multi-user is now supported by icpp-llm and it is again deployed to main-net canister 4c4bn-daaaa-aaaag-abvcq-cai ! It is deployed to main-net, and to see it in action, check out this short loom video , where I run two concurrent shell scripts, and two stories are being generated by the LLM for two…

Hey @icpp , how do you handle randomness at the moment? Is the same prompt giving the same answer or is icpp taking care of this and uses the randomness API of the Internet Computer?

Llama2.c LLM running in a canister!

Developers Programs & Applications

ildefons August 7, 2023, 9:18pm 2

What is the average inference time? If it takes longer than the consensus period, what happens?

Topic		Replies	Views
Introducing the LLM Canister: Deploy AI agents with a few lines of code Developers rust , DeAI	76	4897	September 1, 2025
Llama.cpp on the Internet Computer Programs & Applications	18	738	March 19, 2026
AI and machine learning on the IC? Developers	114	10521	June 20, 2024
How can I take an open source pretrained LLM model, deploy it to ICP and use as a private ChatGPT just fo me Developers	13	495	June 16, 2025
Is llama-13B(or 7B) LLM possible to deploy on canister? Developers Discussing	5	566	October 8, 2025

Llama2.c LLM running in a canister!

Related topics