Technical Working Group DeAI

[icpp]

Hi All,
cross-posting here for awareness. Looking forward to feedback on this pre-release of ICGPT with a llama.cpp backend.

Llama.cpp on the Internet Computer

It’s been a journey, but a pre-release of ICGPT with a llama.cpp backend is now live on the IC. The deployed llama.cpp model is Qwen 2.5 - 0.5B - Q8_0 You can watch a small video here You can try it out at https://icgpt.icpp.world/ A 0.5B model with q8_0 quantization fits fine in a 32bit canister. However, because of the instruction limit, which requires multiple update calls, it takes about 2 minutes to get this answer to the question shown below We did not do any load testing, so it will be …