News for hackers. Stuff that matters.
Re:Llama.cpp 30B runs with only 6GB of RAM now by bee_rider on Friday March 31, 2023 @05:32PM (#35393957)
The professional optimizes well enough to get management off their back, the hobbyist can be irrationally good.
Re:Llama.cpp 30B runs with only 6GB of RAM now by fallous on Friday March 31, 2023 @07:47PM (#35395341)
The professional operates within prioritization tranches. Make it work, make it reliable, make it fast, make it pretty. If you're still iterating on proof-of-concept/prototyping you'll generally confine yourself to the first and/or second levels. Once you've settled on a finalized prototype you then follow the rest of the prioritization levels to achieve shippable product.