Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel
by ndriscoll on Wednesday October 11, 2023 @11:44PM
(#37853287)
Back in the day, Google was an innovator by using lots of cheap commodity servers instead of a few expensive ones and just accepting failures as a fact of life. I wonder 25% seriously whether there's an opportunity for a similar mad genius move to pay for business class fiber at a half dozen remote employee's homes across the country and just have a good replication/failover strategy. 24/7 on call isn't that big of a deal if you just have to go into your basement to swap a drive. Going to be on vacation? Don't be the primary site while you're out.
[ Parent
| on HN ]
Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel
by benreesman on Thursday October 12, 2023 @02:31AM
(#37854092)
People like vast.ai are making moves in this geberal direction FWIW if you’re passionate about it.