You can buy ex-data center hardware today, and folks regularly do. While power is a concern, it's usually not what stops anyone from doing this. What stops most people other than not having enough space is going to be the noise and heat.
Even something as relatively compute-less as a datacenter rack mount ethernet switch often has fans that run at 50-75db, simply because they are designed for environments were silence/low volume isn't a major design consideration like it is for virtually all other consumer electronics.
> https://docs.nvidia.com/dgx-superpod/design-guides/dgx-super...
While there are a ton of factors in determining how loud a given system will be, the linked doc above reckons even a single h100 is typically almost 100db at 80% fan power, with spikes of up to 106 possible for single servers.
That's why I pointed out the next generation is coming soon. Also, the AMD docs aren't using quantization (as far as I can tell, I only skimmed), which gives a speedup roughly linear in the compression ratio. Algorithms for that continue to improve, so expect a lossless factor of 2-8x on DRAM and throughput, at least.
If you take four Framework Desktop 128 GB Strix Halo motherboards at a previous low and currently-unattainable price, and ignore the cost of storage, power, networking, and cases, and then take an older model that's just about competitive with Opus 4.5, and then you quantise that model down to Q2_K_XL and ignore the performance degradation below Opus 4.5 this will cause, and then you compare the cost of this with the highest tier of Claude subscription that includes much newer models which are far more able than your Opus 4.5 benchmark, then you might have some sort of break-even inside a year.
Yes, there's a new AMD hardware generation coming, but it's going to be as heinously expensive as the current one has become, and it's still got relatively low memory bandwidth. Yes, there are already newer models than Kimi 2.5, but with the limitations of this cluster (e.g. still needing heavily quantised models to be feasible) you'll get incremental improvements at best.
I'm keen to be supportive and bullish about open/home models, but I worry that this is such a stretch, and the two options in your comparison are so incomparable, you'll turn far more people off than you convince.
https://www.amd.com/en/developer/resources/technical-article...
At 7 months of claude code subscription per node, the cluster pays for itself in 28 months. On a 5 year (60 month) depreciation schedule, you can buy two of those clusters for basically break even, so you get two concurrent request streams (each of which can batch, etc).
The next generation hardware has already been announced, and should ship roughly two Moore’s law doublings later. It’s likely its steady state price is <= $1400 USD (2024), and it is faster.
So, once the bubble pops (because the financial machinations eventually will come to an abrupt halt), and the labs stop buying hardware for data centers, local inference will be extremely practical and cheaper than a subscription.
My main question is, when that happens, will UNIX Surplus be selling inference servers for pennies on the dollar (like after the dotcom crash), or are the power requirements too exotic for home use?