Re:M2 Ultra can run 128 streams of Llama 2 7B in parallel
by civilitty on Wednesday October 11, 2023 @09:52PM
(#37852600)
In order to reliably get an 8x A100 or 8x H100 pod from AWS you need to get 1 year reserved instances. That 1 year costs significantly more than an equivalent LambdaLabs pod at retail, leaving tens of thousands per pod for hosting and management. Per pod. An engineer breaks even managing a dozen of them!
There’s so much demand companies are renting boxes that they literally pay for with their commitments plus an AWS tax. The numbers are bonkers right now and if you don’t want to go with a reserved instances, on demand is like $40/hr (last I checked). Cloud servers were never that much cheaper but now with GPUs the total cost of rental has exceeded total cost of ownership by a significant amount.
There’s so much demand companies are renting boxes that they literally pay for with their commitments plus an AWS tax. The numbers are bonkers right now and if you don’t want to go with a reserved instances, on demand is like $40/hr (last I checked). Cloud servers were never that much cheaper but now with GPUs the total cost of rental has exceeded total cost of ownership by a significant amount.