News for hackers. Stuff that matters.
Re:Open-weight AI is having its Kubernetes moment. Let's not... by amazingamazing on Saturday July 25, 2026 @01:37PM (#49049727)
Sadly until china scales production of hardware it really isn’t economical to run this stuff yourself. It is good it exists though to put pressure against the labs.

Honestly imo this is just proof apple will win in the end. Eventually a phone will be able to run a model good enough to do most things and it then is game over.

Re:Open-weight AI is having its Kubernetes moment. Let's not... by sschueller on Saturday July 25, 2026 @02:00PM (#49049957)
Model-on-Chip is coming. GPU are for general computing but have a huge bottle neck for doing model inference.

Even not being able to significantly update a model that is burned on a chip the performance gains are immense. You also don't need the latest chip fabs to make them drastically reducing the cost.

Re:Open-weight AI is having its Kubernetes moment. Let's not... by cousinbryce on Saturday July 25, 2026 @04:01PM (#49050997)
This will probably work well with SotA planning and local chip implementation. I see them being like cars. Cost a few 10k on credit, buy a new one when the old one goes bad or marketing convinces you to upgrade.
Re:Open-weight AI is having its Kubernetes moment. Let's not... by baby_souffle on Saturday July 25, 2026 @03:38PM (#49050786)
I don't think asics specific to a specific model or even model family are likely to be commodity hardware anytime soon.

It's extremely expensive to build that and you'll be at least two major model generations behind before you even get your first wafers back. By the time you got your production run ready to go and packaged for market nobody's going to care.

Once we end up going something like 24 months between major advances and capabilities for these models then I can start to see asics for a model being possible.

Re:Open-weight AI is having its Kubernetes moment. Let's not... by thisoneisreal on Saturday July 25, 2026 @03:24PM (#49050685)
One thought I had is that you could use FPGAs to get hardware performance but maintain the ability to dynamically update. I don't know enough about hardware to consider trying such a thing but I'm curious if that could be made practical and economical somehow.
Re:Open-weight AI is having its Kubernetes moment. Let's not... by distantviewer on Saturday July 25, 2026 @05:21PM (#49051716)
Probably not for 2-3 decades unfortunately. The largest FPGAs in the world come out around ~10 million cells (the programmable component of an FPGA). GPT-2 is 1600 million parameters, Qwen 3.5's smallest model is 800 Million parameters.
Re:Open-weight AI is having its Kubernetes moment. Let's not... by nicce on Saturday July 25, 2026 @02:29PM (#49050235)
Many years until consumers can buy them at reasonable price. Nvdia and AMD are making GPUs bad in purpose for consumers so that nobody can build a datacenter from them. It will take a long time.
Re:Open-weight AI is having its Kubernetes moment. Let's not... by Danox on Saturday July 25, 2026 @04:09PM (#49051048)
Hardware and software getting better every day the barbarians are at the gate…
Re:Open-weight AI is having its Kubernetes moment. Let's not... by ben_w on Saturday July 25, 2026 @04:29PM (#49051232)
> Even not being able to significantly update a model that is burned on a chip the performance gains are immense. You also don't need the latest chip fabs to make them drastically reducing the cost.

Yeeeees but the models are in some sense doubling in performance every 4 months, so I expect this to happen in serious quantities approximately when the economic bubble bursts and investors are no longer willing to pay for training.

(Based on widespread news reporting of the existing impact on US electricity markets, I expect this around the end of this year; but with regards to news reporting I am aware of the Gell-Mann amnesia effect, so if this is as much BS as the water issue turned out to be…)

Re:Open-weight AI is having its Kubernetes moment. Let's not... by segmondy on Saturday July 25, 2026 @03:27PM (#49050711)
I don't know what you mean by "economical", but it has been "economical" to run this stuff yourself for the last 3 years.

1. You must be willing to be resourceful. 2. Be willing to learn, do the hard things. 3. Accept the tradeoffs.

Re:Open-weight AI is having its Kubernetes moment. Let's not... by hedora on Saturday July 25, 2026 @03:40PM (#49050821)
Pre bubble prices (~= “we stop building data centers with subsidized credit / circular loans / hidden debt”), a 128GB halo strix ran for $1400, and 200-ish watts. Four of those in a cluster will run a 1T parameter frontier model:

https://www.amd.com/en/developer/resources/technical-article...

At 7 months of claude code subscription per node, the cluster pays for itself in 28 months. On a 5 year (60 month) depreciation schedule, you can buy two of those clusters for basically break even, so you get two concurrent request streams (each of which can batch, etc).

The next generation hardware has already been announced, and should ship roughly two Moore’s law doublings later. It’s likely its steady state price is <= $1400 USD (2024), and it is faster.

So, once the bubble pops (because the financial machinations eventually will come to an abrupt halt), and the labs stop buying hardware for data centers, local inference will be extremely practical and cheaper than a subscription.

My main question is, when that happens, will UNIX Surplus be selling inference servers for pennies on the dollar (like after the dotcom crash), or are the power requirements too exotic for home use?

Re:Open-weight AI is having its Kubernetes moment. Let's not... by giobox on Saturday July 25, 2026 @03:56PM (#49050955)
> or are the power requirements too exotic for home use?

You can buy ex-data center hardware today, and folks regularly do. While power is a concern, it's usually not what stops anyone from doing this. What stops most people other than not having enough space is going to be the noise and heat.

Even something as relatively compute-less as a datacenter rack mount ethernet switch often has fans that run at 50-75db, simply because they are designed for environments were silence/low volume isn't a major design consideration like it is for virtually all other consumer electronics.

> https://docs.nvidia.com/dgx-superpod/design-guides/dgx-super...

While there are a ton of factors in determining how loud a given system will be, the linked doc above reckons even a single h100 is typically almost 100db at 80% fan power, with spikes of up to 106 possible for single servers.

Re:Open-weight AI is having its Kubernetes moment. Let's not... by mft_ on Saturday July 25, 2026 @04:32PM (#49051273)
I’m happy to be proven wrong, but the limited examples I’ve seen of clustered Strix Halos are quite slow running large models (ie models too large to fit into the ram of a single machine) due to the slow networking between each one?
Re:Open-weight AI is having its Kubernetes moment. Let's not... by downrightmike on Saturday July 25, 2026 @09:16PM (#49053622)
That is already being addressed with GPUs and special PCIe daughter cards that link them together faster than the PCIe on the MOBO could. SLI but better.
Re:Open-weight AI is having its Kubernetes moment. Let's not... by hedora on Saturday July 25, 2026 @10:52PM (#49054198)
Yeah; the benchmarks + setup guide I linked show that it takes a lot of careful setup to get them to work at all, and then you only get 100's of tokens per second per cluster.

That's why I pointed out the next generation is coming soon. Also, the AMD docs aren't using quantization (as far as I can tell, I only skimmed), which gives a speedup roughly linear in the compression ratio. Algorithms for that continue to improve, so expect a lossless factor of 2-8x on DRAM and throughput, at least.

Re:Open-weight AI is having its Kubernetes moment. Let's not... by mft_ on Sunday July 26, 2026 @06:29AM (#49056603)
If I rephrase your earlier post, and based on the AMD post:

If you take four Framework Desktop 128 GB Strix Halo motherboards at a previous low and currently-unattainable price, and ignore the cost of storage, power, networking, and cases, and then take an older model that's just about competitive with Opus 4.5, and then you quantise that model down to Q2_K_XL and ignore the performance degradation below Opus 4.5 this will cause, and then you compare the cost of this with the highest tier of Claude subscription that includes much newer models which are far more able than your Opus 4.5 benchmark, then you might have some sort of break-even inside a year.

Yes, there's a new AMD hardware generation coming, but it's going to be as heinously expensive as the current one has become, and it's still got relatively low memory bandwidth. Yes, there are already newer models than Kimi 2.5, but with the limitations of this cluster (e.g. still needing heavily quantised models to be feasible) you'll get incremental improvements at best.

I'm keen to be supportive and bullish about open/home models, but I worry that this is such a stretch, and the two options in your comparison are so incomparable, you'll turn far more people off than you convince.

Re:Open-weight AI is having its Kubernetes moment. Let's not... by JumpCrisscross on Saturday July 25, 2026 @01:53PM (#49049890)
> it really isn’t economical to run this stuff yourself

Quantised models running overnight go most of the way for non-coding tasks.

Re:Open-weight AI is having its Kubernetes moment. Let's not... by root-parent on Saturday July 25, 2026 @01:47PM (#49049825)
>> do most things and it then is game over.

For the Hyperscalers...and Oracle...cant wait for the day...

Re:Open-weight AI is having its Kubernetes moment. Let's not... by esseph on Saturday July 25, 2026 @03:19PM (#49050660)
And it floods the market with millions looking for work
Re:Open-weight AI is having its Kubernetes moment. Let's not... by esseph on Saturday July 25, 2026 @03:15PM (#49050629)
> Sadly until china scales production of hardware it really isn’t economical to run this stuff yourself.

I'm running this stuff at home on my desktop and using it through an app on my phone. 60-140TPS depending on model / use case.

It's more than fast enough to even maintain voice conversation.

Re:Open-weight AI is having its Kubernetes moment. Let's not... by esseph on Saturday July 25, 2026 @03:18PM (#49050654)
> Honestly imo this is just proof apple will win in the end. Eventually a phone will be able to run a model good enough to do most things and it then is game over.

I don't see how these are related.

The accuracy and capabilities of your model are directly related to its size. You need a lot of memory for that.

It will be decades before we get enough useful memory in a phone form factor at a price point people can afford it before something like a frontier model now is useful on the phone.

Now, you can run some models on your phone today.

Either way, Apple is using Google today. That could change, but Google isn't exactly getting out of the TPU business and they've been doing it a long time.

Also, some of you live in a very weird Apple bubble. Apple is not so relevant outside the US.

Re:Open-weight AI is having its Kubernetes moment. Let's not... by Danox on Saturday July 25, 2026 @04:51PM (#49051432)
If it wasn’t for the current memory fiasco, Apple would be even closer to providing a solution on the desktop that is affordable for most (Hacker News participants) who want to run a large model locally at reasonable cost, which will happen within the next 3 to 5 years easy with the way software and hardware are progressing, the main-frame future for most people that use computers isn’t coming back so OpenAI and Anthropic and other frontier developers are going to be disappointed and will have to shift gears ala’ Meta, Microsoft?

The United States if it dares to (I think they will try) but isn’t going to be able to stuff AI models back into the bottle open source is the future and when it comes to AI models yes you’ll be able to customize it to your specifications locally but the genie is out of the bottle. The bull out of the barn and is running down the road.

If United States insist on trying to lock the doors, censor, sanction, the rest of the world will just design and engineer around the United States. Trying to put up a wall, will damaged the United States more particularly with the current performance of Taco. None of the other countries are going to follow the United States not with the current administration they will hedge their bets.

Apple, is using Google now but that will change because the world is probably going down the open path, it’s looking like there was no real rush and no reason to spend so much money on something that’s going to be a commodity in the end, the only hold up is hardware and if it wasn’t for this current memory fiasco, many more people would have access to the hardware that they need to run models locally.

Re:Open-weight AI is having its Kubernetes moment. Let's not... by esseph on Saturday July 25, 2026 @06:20PM (#49052263)
> If it wasn’t for the current memory fiasco, Apple would be even closer to providing a solution on the desktop that is affordable for most (Hacker News participants)

Already running models locally without apple hardware, and have an encrypted vpn tunnel from my mobile devices back to my desktop over the internet. The time is now.