Re:Llama2.c: inference llama 2 in one file of pure C
by Y_Y on Sunday July 23, 2023 @04:31PM
(#36839510)
It might be more expensive to get a GPU instance but at a guess I'd say it's more cost-effective considering that the CPU computation will be less efficient and take much longer. I bet someone's done this out with real numbers, I just haven't seen it.
[ Parent
| on HN ]
Re:Llama2.c: inference llama 2 in one file of pure C
by franga2000 on Sunday July 23, 2023 @06:21PM
(#36840505)
This only matters if you're scaling to meet demand and demand is higher than your spare resources, which often isn't the case for hobby projects.
The 10€/mo VPS I've had for over 6 years now still has a few cores and GBs or RAM spare, so running a small model on the CPU for a personal project that only me and a few friends occasionally use wouldn't cost me a cent more.