Re:Llama.cpp 30B runs with only 6GB of RAM now
by dplavery92 on Friday March 31, 2023 @06:09PM
(#35394356)
Sure, but when one 12gb GPU costs ~$800 new (e.g. for the 3080 LHR), "a couple of dozens" of them is a big barrier to entry to the hobbyist, student, or freelancer. And cloud computing offers an alternative route, but, as stated, distribution introduces a new engineering task, and the month-to-month bills for the compute nodes you are using can still add up surprisingly quickly.
[ Parent
| on HN ]