I want everything local – Building my offline AI workspace

Re:I want everything local – Building my offline AI workspace by ethan_smith on Saturday August 09, 2025 @12:49PM (#44848023)

For Intel CPUs, Phi-2 (2.7B) and TinyLlama (1.1B) run reasonably well using llama.cpp with 4-bit quantization. GGUF models with INT4 quantization typically need ~2GB RAM per billion parameters, so even older machines can handle smaller models.

[ Parent | on HN ]

Re:I want everything local – Building my offline AI workspace by akawry on Sunday August 10, 2025 @09:23AM (#44855027)

Take a look at ik_llama.cpp: https://github.com/ikawrakow/ik_llama.cpp

CPU performance is much better than mainline llama, as well as having more quantization types available

[ Parent | on HN ]