News for hackers. Stuff that matters.

A comment on M2 Ultra can run 128 streams of Llama 2 7B in parallel. Parent comment

Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by mannyv on Wednesday October 11, 2023 @08:34PM (#37852134)
The problem with using a cloud provider is that you still need to know what you're doing.

Your application isn't going to magically become HA/DR. You still have to make it that way, from your application design/coding up through the deployment.

I mean, if you're not storing your session IDs in a data store that's reachable by all the nodes behind your load balancer then no amount of infrastructure is going to save you.

[ Parent | on HN ]
Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel by TheNewsIsHere on Thursday October 12, 2023 @10:27AM (#37857655)
A great illustration of this is the GitHub outage a few years back. They had a fairly well distributed application layer but the database topography at the time didn’t consider the failure mode, even though the application layer did.

That’s a realistic scenario no matter whether you’re bare metal, building out your own cloud, or using someone else’s. No amount of AWS/GCP/Azure/et al marketing changes that.

[ Parent | on HN ]