Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Ask HN: What is your local LLM setup?.
- B550MH motherboard
- Ryzen 3 4100 CPU
- 32GB (2x16) RAM cranked up to 3200MHz (prompt generation in memory bound)
- 256GB M.2 NVMe (helps with loading models faster)
- Nvidia 3060 12GB
Software-wise, I use llamafile because on the CPU it's faster by 10-20% for prompt processing than llama.cpp.
Performance "Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf":
CPU-only: 23.47 t/s (processing), 8.73 t/s (generation)
GPU: 941.5 t/s (processing), 29.4 t/s (generation)
There are no comments on this comment yet.
- B550MH motherboard
- Ryzen 3 4100 CPU
- 32GB (2x16) RAM cranked up to 3200MHz (prompt generation in memory bound)
- 256GB M.2 NVMe (helps with loading models faster)
- Nvidia 3060 12GB
Software-wise, I use llamafile because on the CPU it's faster by 10-20% for prompt processing than llama.cpp.
Performance "Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf":
CPU-only: 23.47 t/s (processing), 8.73 t/s (generation)
GPU: 941.5 t/s (processing), 29.4 t/s (generation)