News for hackers. Stuff that matters.
Re:Llama2.c: inference llama 2 in one file of pure C by Y_Y on Sunday July 23, 2023 @04:31PM (#36839510)
It might be more expensive to get a GPU instance but at a guess I'd say it's more cost-effective considering that the CPU computation will be less efficient and take much longer. I bet someone's done this out with real numbers, I just haven't seen it.
Re:Llama2.c: inference llama 2 in one file of pure C by franga2000 on Sunday July 23, 2023 @06:21PM (#36840505)
This only matters if you're scaling to meet demand and demand is higher than your spare resources, which often isn't the case for hobby projects. The 10€/mo VPS I've had for over 6 years now still has a few cores and GBs or RAM spare, so running a small model on the CPU for a personal project that only me and a few friends occasionally use wouldn't cost me a cent more.
Re:Llama2.c: inference llama 2 in one file of pure C by immibis on Monday July 24, 2023 @02:30AM (#36843784)
FYI, the going rate for "smallest possible VPS" is now more like 3€/mo.
Re:Llama2.c: inference llama 2 in one file of pure C by bg24 on Sunday July 23, 2023 @07:36PM (#36841164)
It depends on your use case, correct? If you do not have a heavy inferencing requirement, then CPU is good enough.