Re:OpenAI Needs $400B In The Next 12 Months
by edude03 on Friday October 17, 2025 @04:59PM
(#45621993)
$50k would be the cost to run it un-quantized, 10k could get you for example 4 5090 system, that would run the 671b q4 model which is 90% as good, which was the OPs target
Re:OpenAI Needs $400B In The Next 12 Months
by tkz1312 on Friday October 17, 2025 @06:10PM
(#45622732)
which 671b quants can fit into 96GB VRAM? Everything I’m aware of needs hundreds at least (e.g. https://apxml.com/models/deepseek-r1-671b).
Re:OpenAI Needs $400B In The Next 12 Months
by lossolo on Friday October 17, 2025 @07:26PM
(#45623339)
5090 is 32 GB so it's 128GB, not 96.
Re:OpenAI Needs $400B In The Next 12 Months
by tkz1312 on Friday October 17, 2025 @11:30PM
(#45624674)
128 is still not 300. Something like 4x 6000 blackwell is the minimum to run any model that is going to feel anything like claude locally.
To my deep disappointment the economics are simply not there at the moment. Openrouter using only providers with zero data retention policies is probably the best option right now if you care about openness, privacy and vendor lock-in.
Re:OpenAI Needs $400B In The Next 12 Months
by 999900000999 on Saturday October 18, 2025 @02:30PM
(#45629412)
For local use and experimentation you don't need to match a top of the line model. In fact something that you train or rather fine-tune locally might be better for certain use cases.
If I was working with sensitive data I sure would only use on prem models.
The hardware required to run something like deepseek / kimi / glm locally at any speed fast enough for coding is probably around $50,000. You need hundreds of gigabytes of fast VRAM to run models that can come anywhere close to openai or anthropic.