What graphics cards have more than 12GB, and how much does it cost to buy one? What about for others saying you ideally want 32GB? (That split will determine if a 5080 will do it)

32GB would probably get you qwen3.8-27b at a reasonable quantisation, which benchmarks similar to opus 4.6

You can try that model (and many others) using a cloud host first, and then only if you like it do you need to buy something to run it.

> What graphics cards have more than 12GB, and how much does it cost to buy one?

RTX 5070 Ti and RTX 5080 both have 16 GB.

RTX 5090 has 32 GB.

Good luck finding any of them for reasonable prices, though. I paid $3,000 for my 5090 early last year. I winced when I bought it then. Today, it's like $7,000, and I wouldn't even consider it at that price.

But I'm mostly a gamer that also does local AI as a hobby. If the RTX 5090 was $7,000, I'd still be on my RTX 3080 and still just not started doing local AI and have to turn down the graphics settings in Cyberpunk 2077.

there is also a version of the 5060ti woth 16GB. i got lucky a few months ago and got it for ~380€, but i think you can still get it for around 800, definitely under 1k

The RTX 5090 is the ideal, but it's not the only 32GB GPU that can offer workable performance; the AMD R9700 is catching up.

You can surely build an LLM server with an R9700 and 32GB of mainboard RAM for less than the 64GB M5 Max Apple Studio, which is probably the next most competitive model up. It'll be better than a Strix Halo but less flexible.

Of interest:

  - 4090: Going for $7500
  - 64GB M5 Max: Starts at $3,800
  - R9700: $2700

Compared to $20 or $100 per month.

Right. Nobody says it's necessarily economically the best option.

But it's also an asset that barely depreciates!

Especially right now — you could buy and extensively use an M5 Max for a year and sell it for almost exactly the same amount at the end. Maybe even more, at the current rate.

Furthermore, it can do other things for you: host your VMs, edit high def video, whatever. Even if it stops being useful to serve LLMs, it might be useful to run as a server for other tasks for a whole decade.

Depends what you want, doesn't it? Or are we all expected to conform?

For me personally, right now I have an M1 Max macbook pro, and that is my local AI machine, because my needs are exploratory and I want to run it here. I want to learn by comparing models. I want to learn how to run them and advise others how to (local inference is not a zero-sized market).

My next step up will be to take the apps I'm looking at building that run against these small models, and host those apps online with some sort of cloud inference — openrouter, digitalocean, cloudflare. Again — if I can avoid it, my apps won't develop a dependency on any AI provider.

If I ever find I want to vaguely describe some desktop application before bed and come down the next morning to an aggravating slop version of it, I'll consider Claude.

As for a local machine, I've looked at the M5 Max and the R9700; haven't pushed the button yet because I don't need it enough.

I've also looked at the M5 Pro, which in practice is about as fast at token generation as my M1 Max, and smokes it at prefill. That alone might be enough for a "leave it running" mini AI box, because agentic coding is so prefill-heavy. But I am a laptop person for the next six months or so I think.

As an aside there's that other thing which is somewhat interesting, though constrained:

https://www.olares.com/one

Uses the 5090 mobile chip, which is pretty good speedwise (about the same as the R9700) but 24GB. Small, tidy, on-prem. I think they have missed the boat on pricing, but if they can ship in volume, maybe not.

Great insights! And I'm kicking myself for not buying a 4090 3 years ago for $1600, shipped direct from nvidia... it seemed like too much to pay for a GPU at the time!

I know a couple of people who have eminently LLM-capable GPUs more or less doing nothing when they are at work...

The assumption that is probably incorrect is that the current prices will last. A 4090 should not cost that much, and most of your Claude usage is paid by an investor. Neither of these things will be true in a few years.

Having a 4090 at home running agents all day is not a $20pcm level of usage, so take the $100 plan. Looking at the S-1 filing, today they are not even close to breakeven. It is reasonable to assume that will become a $300-400 pcm plan.

The launch MSRP of a 4090 was $1600. There has been significant inflation since then, so say $2000. Eventually supply will meet demand and you'll be able to buy one for that price again, or lower due to the economies of scale they must have now.

So, in 2-3 years time once it has all settled, you pay off your 4090 by saving 5-6 months of Claude usage. Maybe the local model isn't good enough for complete replacement, but all it needs to do is drop you to the $20 plan to be worth it. I think this is a large part of why investors are getting jittery now.