Re:Llama.cpp 30B runs with only 6GB of RAM now
by bee_rider on Friday March 31, 2023 @05:32PM
(#35393957)
The professional optimizes well enough to get management off their back, the hobbyist can be irrationally good.
[ Parent
| on HN ]
Re:Llama.cpp 30B runs with only 6GB of RAM now
by fallous on Friday March 31, 2023 @07:47PM
(#35395341)
The professional operates within prioritization tranches. Make it work, make it reliable, make it fast, make it pretty. If you're still iterating on proof-of-concept/prototyping you'll generally confine yourself to the first and/or second levels. Once you've settled on a finalized prototype you then follow the rest of the prioritization levels to achieve shippable product.