News for hackers. Stuff that matters.

A comment on Running Llama.cpp on AWS Instances.

Re:Running Llama.cpp on AWS Instances by mikeravkine on Tuesday November 28, 2023 @09:11AM (#38445852)
If anyone is looking for a more reasonably cost effective solution, Hetzner has 16 vCPU/32GB RAM ARM VMs for $24E/mo that will run 34b Q4 GGUF at around 4 tok/sec. It's not very fast, but it is very cheap.
[ Parent | on HN ]

There are no comments on this comment yet.