Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel
by TheNewsIsHere on Thursday October 12, 2023 @10:27AM
(#37857655)
A great illustration of this is the GitHub outage a few years back. They had a fairly well distributed application layer but the database topography at the time didn’t consider the failure mode, even though the application layer did.
That’s a realistic scenario no matter whether you’re bare metal, building out your own cloud, or using someone else’s. No amount of AWS/GCP/Azure/et al marketing changes that.
That’s a realistic scenario no matter whether you’re bare metal, building out your own cloud, or using someone else’s. No amount of AWS/GCP/Azure/et al marketing changes that.