### Author:  [@icpp](https://forum.dfinity.org/u/icpp)
#### Post date: [July 21, 2024, 4:30pm UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/1 "2024-07-21T16:30:53Z")
This thread discusses **llama.cpp on the Internet Computer**.
A project funded by the DFINITY Grant: **ICGPT V2**
The first functioning version is now MIT licensed open source:
> **[GitHub - onicai/llama\_cpp\_canister: llama.cpp on the Internet Computer](https://github.com/onicai/llama_cpp_canister)**
>
> llama.cpp on the Internet Computer. Contribute to onicai/llama\_cpp\_canister development by creating an account on GitHub.
---
### Author:  [@icpp](https://forum.dfinity.org/u/icpp)
#### Post date: [July 21, 2024, 4:31pm UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/2 "2024-07-21T16:31:19Z")
Current status:
- on a Mac, you can build/deploy/upload the LLM, and then call an endpoint to have it generate tokens.
- it only works for a small model, because no use is made yet of the recent advancements of the IC (SIMD, float handling, etc.).
- the README of the GitHub repo contains a list of TODOs
---
### Author:  [@icpp](https://forum.dfinity.org/u/icpp)
#### Post date: [October 15, 2024, 6:21pm UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/3 "2024-10-15T18:21:13Z")
It’s been a journey, but a pre-release of ICGPT with a llama.cpp backend is now live on the IC.
- The deployed llama.cpp model is Qwen 2.5 - 0.5B - Q8\_0 - Instruct
- You can watch a small video [here](https://youtu.be/-5uBiFMa1ec?feature=shared)
- You can try it out at [https://icgpt.icpp.world/](https://icgpt.icpp.world/)
- A 0.5B model with q8\_0 quantization fits fine in a 32bit canister.
- However, because of the instruction limit, which requires multiple update calls, it takes about 2 minutes to get this answer to the question shown below
- We did not do any load testing, so it will be interesting to see how it holds up when multiple users try it out at the same time.
- The UI is still primitive. The same as the one that was developed for the tiny story teller LLM. Improving that is on the to-do list.
---
### Author:  [@icpp](https://forum.dfinity.org/u/icpp)
#### Post date: [October 30, 2024, 4:47am UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/4 "2024-10-30T04:47:58Z")
🎉ICGPT V2 - final milestone reached 🎉
_(I also posted this on [X](https://x.com/icpp_pro/status/1851482949520375835))_
The grant work is now completed and here is a video summarizing what I created. I want to thank [@dfinity](https://x.com/dfinity) for the opportunity & the support.
I also want to thank the [#ICP](https://x.com/hashtag/ICP?src=hashtag_click) community for the enthusiasm as I shared progress along the way over the past months, and the testing some of you did with the early releases.
Some of you even donated cycles that will keep the Qwen2.5 canister up & running for several months. You are the best. 🙂
You can try it out at: [https://icgpt.icpp.world](https://icgpt.icpp.world)
I am very happy with the outcome of this project and there are big plans to build on top on this foundation. More on this later. But first some time to celebrate this milestone 🍾🍾
[Youtube Video](https://youtu.be/oAgAgoMgITM)
---
### Author:  [@paulous](https://forum.dfinity.org/u/paulous)
#### Post date: [October 30, 2024, 5:49am UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/5 "2024-10-30T05:49:11Z")
Worked better this time 💪
---
### Author:  [@superduper](https://forum.dfinity.org/u/superduper)
#### Post date: [November 7, 2024, 11:52pm UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/6 "2024-11-07T23:52:16Z")
congrats super cool i hope this gets the use and attention it deserves
btw i’m playing with a solana project [https://github.com/ai16z/eliza/](https://github.com/ai16z/eliza/) which can connect to to remote or local LLMs
does this have API end points so that we could add it as one of the remote models? i can only pidgeon code so hard for me to eval how it works
>
---
### Author:  [@icpp](https://forum.dfinity.org/u/icpp)
#### Post date: [November 8, 2024, 12:25pm UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/7 "2024-11-08T12:25:59Z")
@superduper ,
thanks for your feedback and for pointing out the eliza model. I will check it out.
About the API:
- There are two endpoints [new\_chat](https://github.com/onicai/llama_cpp_canister/blob/c2208289a71c036b92abd645b24e2cb68437a207/src/llama_cpp.did#L105) & [run\_update](https://github.com/onicai/llama_cpp_canister/blob/c2208289a71c036b92abd645b24e2cb68437a207/src/llama_cpp.did#L107). These are candid based canister endpoints. The links bring you to the candid service definition. In the [README](https://github.com/onicai/llama_cpp_canister/tree/main) it is described how you call it using dfx. This API is using the llama.cpp command line interface, and makes it really easy to test things locally, and then use the same arguments when calling the canister.
- We’re looking at creating another API that is following the openAI completions standard.
---
### Author:  [@roger](https://forum.dfinity.org/u/roger)
#### Post date: [November 22, 2024, 4:50pm UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/8 "2024-11-22T16:50:44Z")
Congratulations on the final milestones! Must have been some journey. 😀
We just got done with our first milestone of LLM Marketplace and will be getting into an interesting phase. In this phase I was planning to explore Tiny llama (1.1 B parameters) and train it for a niche task.
But looking at your post I’m having second thoughts, primarily because your model is smaller than tiny llama and quantised. Despite, it hits the instruction limit restriction. I will experiment and share my learning.
I’m trying to brainstorm ideas to see what could be a tiny task it could be trained on bypassing the instruction limit.
Would appreciate your thoughts around it.
Cheers!
---
### Author:  [@icpp](https://forum.dfinity.org/u/icpp)
#### Post date: [November 22, 2024, 8:38pm UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/9 "2024-11-22T20:38:07Z")
Hi @roger ,
The 1.1B Tiny Llama will not fit.
I recommend you select a 0.5B parameter model, like the Qwen 2.5 model I am using, and try to fine tune that one.
---
### Author:  [@roger](https://forum.dfinity.org/u/roger)
#### Post date: [November 23, 2024, 11:21am UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/10 "2024-11-23T11:21:37Z")
Yes, I have been contemplating to use Qwen and also exploring few other light wt models like SmolLM, DistilGpt2.
While researching these in also trying to close on a fun use case.
Thank you for the suggestion.
---
### Author:  [@icpp](https://forum.dfinity.org/u/icpp)
#### Post date: [February 2, 2025, 3:16pm UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/11 "2025-02-02T15:16:27Z")
We completed the update to the latest llama.cpp version (sha 615212).
Please start fresh by following the instructions at [GitHub - onicai/llama\_cpp\_canister: llama.cpp for the Internet Computer](https://github.com/onicai/llama_cpp_canister)
This update allows you run many new LLM architectures, including the 1.5Billion parameter DeepSeek model that attracted a lot of attention with this [X post](https://x.com/onicaiHQ/status/1884339580851151089).
The main limiting factor of running the larger LLMs is the instructions limit. If a model can generate at least 1 token, you can use it, because we generate tokens via multiple update calls. (See the README in the repo for details.)
Latency is off course high, which hopefully will improve with further ICP protocol and hardware updates, but we believe it is already possible to build useful, targeted AI agents with their LLM running on-chain. It requires some smart prompt engineering, and this is an area where we are focusing our efforts.
To assist with prompt engineering, a python notebook [prompt-design.ipynb](https://github.com/onicai/llama_cpp_canister/blob/main/scripts/prompt-design.ipynb) is included in the repository, where you can run against the original llama.cpp compiled for your native system.
---
### Author:  [@icpp](https://forum.dfinity.org/u/icpp)
#### Post date: [February 2, 2025, 3:38pm UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/12 "2025-02-02T15:38:46Z")
> [@icpp](#):
>
> We completed the update to the latest llama.cpp version (sha 615212)
A few notes on the testing we did with DeepSeek.
We tested this DeepSeek model, available on HuggingFace:
- [unsloth/DeepSeek-R1-Distill-Qwen-1.5B-GGUF · Hugging Face](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-1.5B-GGUF)
- the Q2\_K model: DeepSeek-R1-Distill-Qwen-1.5B-Q2\_K.gguf
The [model card](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-1.5B-GGUF) on Huggingface shows this llama.cpp command:
```bash
./llama.cpp/llama-cli \
--model unsloth/DeepSeek-R1-Distill-Qwen-1.5B-GGUF/DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf \
--cache-type-k q8_0 \
--threads 16 \
--prompt '<｜User｜>What is 1+1?<｜Assistant｜>' \
-no-cnv
```
Our initial tests confirmed that the parameter --cache-type-k q8\_0 is important to get a good answer from the Q2\_K quantized model.
To call the canister, you would use something like this:
```bash
dfx canister call llama_cpp run_update '(record { args = vec {"--cache-type-k"; "q8_0"; "--prompt-cache"; "prompt.cache"; "--prompt-cache-all"; "-sp"; "-p"; "<｜User｜>What is 1+1?<｜Assistant｜>"; "-n"; "512" } })'
```
_(No need to pass -no-cnv, because that is a default for llama\_cpp\_canister)_
You can generate 2 tokens per update call, so you configure the LLM with this call (Details are in README of repo):
```auto
dfx canister call llama_cpp set_max_tokens '(record { max_tokens_query = 2 : nat64; max_tokens_update = 2 : nat64 })'
```
And that’s really it to get going with this model.
---
### Author:  [@icpp](https://forum.dfinity.org/u/icpp)
#### Post date: [March 19, 2026, 12:49pm UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/13 "2026-03-19T12:49:03Z")
Hi all,
I have picked up work again on llama\_cpp\_canister and the icpp-pro C++ CDK that is an enabler of it.
Since last post in this thread (13 months ago) we successfully used llama\_cpp\_canister with the Qwen2.5-05b-instruct model inside the funnAI application, and it is running smoothly.
Just released llama\_cpp\_canister v0.9.0, where we upgraded the model upload scripts from ic-py to the latest icp-py-core 2.3.0
I will post updates in this thread.
If you have feature requests for either the C++ CDK or llama\_cpp\_canister, let me know.
---
### Author:  [@Henry\_Suso](https://forum.dfinity.org/u/Henry_Suso)
#### Post date: [March 19, 2026, 1:11pm UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/14 "2026-03-19T13:11:29Z")
This is great!
I have recently patched your code to handle upstream llama cpp and bitnet i2s
We can run some .5b .8b models really efficiently in chat formats, anything higher gets caught up in instruction limits.
The best we have running rn are Falcon H1 .5b and Qwen 3.0 in our production.
In our test canister we have: Falcon E 1b i2s instruct running and Qwen 3.0 .6b running. Bitnet 158 large also runs, but it’s not tuned yet and is a base model so no good for chatml. We are basicly working on some edge cases before we move Falcon E 1b to production in our frontend for users to play with.
Our mail goal is to depreciate the Dfinity LLM canister which I believe is off chain? And hook our agent toolchain into Falcon E or some other comparable model.
I’ll be pushing the v5 wasm and the bitnet i2s wasm to this repo shortly
> **[GitHub - Smugandcomfy/icllm: v2 canister interfaces for Small Language Models...](https://github.com/Smugandcomfy/icllm/tree/main)**
>
> main
---
### Author:  [@icpp](https://forum.dfinity.org/u/icpp)
#### Post date: [March 19, 2026, 3:06pm UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/15 "2026-03-19T15:06:41Z")
@Henry_Suso
Very cool. Are you planning to create a PR back into llama\_cpp\_canister or forking off?
btw… I am fine either way – great that you’ve picked this up, and I will be more than happy to support from C++ side.
There are some things I want to add to the C++ CDK in support of running AI workloads and other high compute applications.
---
### Author:  [@icpp](https://forum.dfinity.org/u/icpp)
#### Post date: [March 19, 2026, 3:07pm UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/16 "2026-03-19T15:07:57Z")
> [@Henry\_Suso](#):
>
> the Dfinity LLM canister which I believe is off chain?
Yes, that is indeed off chain. I do not know where the off chain LLM is running.
---
### Author:  [@icpp](https://forum.dfinity.org/u/icpp)
#### Post date: [March 19, 2026, 3:15pm UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/17 "2026-03-19T15:15:39Z")
> [@Henry\_Suso](#):
>
> We can run some .5b .8b models really efficiently in chat formats, anything higher gets caught up in instruction limits
Yes, that was also my experience. I did some detailed studies and summarized it in this table. The biggest one we could fit was Deepseek R1 with 1.78B parameters, but it can only produce 2-4 tokens before hitting the instructions limit – pending on the level of quantization.
* * *
# Appendix A: max\_tokens
The size and settings for models impact the number of tokens that can be generated
in 1 update call before hitting the instruction limit of the Internet Computer.
The instruction limit is 40 billion instructions per update call
We tested several LLM models available on HuggingFace:
| Model | # weights | file size | quantization | –cache-type-k | max\*tokens
\*(ingestion)\_ | max\*tokens
\*(generation)\_ |
| --- | --- | --- | --- | --- | --- | --- |
| [SmolLM2-135M-Instruct-Q8\_0.gguf](https://huggingface.co/tensorblock/SmolLM2-135M-Instruct-GGUF) | 135 M | 0.15 GB | q8\_0 | f16 | - | 40 |
| [qwen2.5-0.5b-instruct-q4\_k\_m.gguf](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF) | 630 M | 0.49 GB | q4\_k\_m | f16 | - | 14 |
| [qwen2.5-0.5b-instruct-q8\_0.gguf](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF) | 630 M | 0.68 GB | q8\_0 | q8\_0 | - | 12 |
| [Llama-3.2-1B-Instruct-Q4\_K\_M.gguf](https://huggingface.co/unsloth/Llama-3.2-1B-Instruct-GGUF) | 1.24 B | 0.81 GB | q4\_k\_m | q5\_0 | 5 | 4 |
| [qwen2.5-1.5b-instruct-q4\_k\_m.gguf](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-GGUF) | 1.78 B | 1.10 GB | q4\_k\_m | q8\_0 | - | 3 |
| [DeepSeek-R1-Distill-Qwen-1.5B-NexaQuant.gguf](https://huggingface.co/NexaAIDev/DeepSeek-R1-Distill-Qwen-1.5B-NexaQuant) | 1.78 B | 1.34 GB | NexaQuant-4Bit | f16 | 4 | 3 |
| [DeepSeek-R1-Distill-Qwen-1.5B-Q6\_K.gguf](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-1.5B-GGUF) | 1.78 B | 1.46 GB | q6\_k | q8\_0 | 4 | 3 |
| [DeepSeek-R1-Distill-Qwen-1.5B-Q4\_K\_M.gguf](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-1.5B-GGUF) | 1.78 B | 1.12 GB | q4\_k\_m | q8\_0 | 4 | 3 |
| [DeepSeek-R1-Distill-Qwen-1.5B-Q2\_K.gguf](https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-1.5B-GGUF) | 1.78 B | 0.75 GB | q2\_k | q8\_0 | 2 | 2 |
---
### Author:  [@Henry\_Suso](https://forum.dfinity.org/u/Henry_Suso)
#### Post date: [March 19, 2026, 4:51pm UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/18 "2026-03-19T16:51:16Z")
I can do that. If I’m honest I am not very smart and don’t really know how to do it. I will try to make a pr!
---
### Author:  [@Henry\_Suso](https://forum.dfinity.org/u/Henry_Suso)
#### Post date: [March 19, 2026, 4:57pm UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/19 "2026-03-19T16:57:15Z")
This is very cool! My current v5 wasm I have qwen3.0 .6b running at 22 tokens and smollm2 360mb running at 50 tokens
---
### Author:  [@icpp](https://forum.dfinity.org/u/icpp)
#### Post date: [July 28, 2026, 9:57pm UTC](https://forum.dfinity.org/t/llama-cpp-on-the-internet-computer/33471/20 "2026-07-28T21:57:59Z")
I want to provide an update on some pretty exciting recent upgrades to [llama\_cpp\_canister](https://github.com/onicai/llama_cpp_canister).
- We upgraded the vendored llama.cpp version to the latest availabe
- That provided a big speedup (2.8x) → See also [https://forum.dfinity.org/t/on-chain-llm-inference-under-instruction-budgets-measured-live-on-icp-mainnet/74709?u=icpp](https://forum.dfinity.org/t/on-chain-llm-inference-under-instruction-budgets-measured-live-on-icp-mainnet/74709)
- Instructions are available to run these models in a canister: qwen3-0.6B, qwen3-1.7B
- And also for a very small model: gemma3-270M
Because these upgrades make it possible to do some interesting things, but you really need to optimize your prompts, we upgraded ICGPT to become an `on-chain LLM prompt studio`
To understand what it does, please read this example from the [docs](https://icgpt.onicai.com/docs/how-it-works#worked-example).
The access to ICGPT is currently gated to early access because it costs significant cycles to run these prompt and we only want to provide access to those of you who have a real use case and want to optimize your prompts.
We have a lot of ideas to build this out further, so I hope some of you will request early access, describe your use case right there in the app, and then join the onicai OpenChat to further discuss your use case.