Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel
by ndriscoll on Wednesday October 11, 2023 @11:44PM
(#37853287)
Back in the day, Google was an innovator by using lots of cheap commodity servers instead of a few expensive ones and just accepting failures as a fact of life. I wonder 25% seriously whether there's an opportunity for a similar mad genius move to pay for business class fiber at a half dozen remote employee's homes across the country and just have a good replication/failover strategy. 24/7 on call isn't that big of a deal if you just have to go into your basement to swap a drive. Going to be on vacation? Don't be the primary site while you're out.
Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel
by benreesman on Thursday October 12, 2023 @02:31AM
(#37854092)
People like vast.ai are making moves in this geberal direction FWIW if you’re passionate about it.
Yes, you have to learn things to goto the cloud, and I won't say it is all roses, it ain't. But... AWS is less likely to fsck it up.
If you have the constant load to burn the flops 24x7x365... go for it. If you have the ops team to do it... go for it.
If you don't... take a bit of time and learn the cloud which is much easier than getting on-prem right.
Especially for smaller firms, this isn't even a close call IMHO.