News for hackers. Stuff that matters.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by mchusma on Tuesday August 25, 2026 @03:44PM (#49439562)
I think they talked about this being general purpose chip but I would think that Anthropic/OpenAI are at the scale now they could bake LLM weights into chips themselves.

For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough.

While 2 years ago nothing was useful more than 1 year long, there are many older models in use now (e.g. Haiku 4.5, GPT-OSS 120b), and I expect this trend to continue.

I know this is what Taalas was doing (acquired by AMD), here was their demo, https://chatjimmy.ai/ which is based on Llama 3.1 8B. It feels like this should start to happen soon.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by bmulholland on Tuesday August 25, 2026 @03:51PM (#49439668)
Probably! But not viable yet; the chips would be about a year behind SOTA. Note the ~16 months that the article quotes as being insanely fast to get this chip to tape-out (read: start producing). We'll have to bootstrap our way there: AI is actively being used to get us closer to viable lead times for this.

Unfortunately, there's some real physical constraints: IIRC, manufacturing a wafer takes on the order of a month, start to finish, for the physical processing.

Maybe once LLM improvements asymptote further?

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by kushie on Tuesday August 25, 2026 @03:59PM (#49439774)
tapeout could shrink but days per mask layer (DPML) does not have much margin..
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by smallmancontrov on Tuesday August 25, 2026 @09:56PM (#49443240)
I'm not in industry, is DPML (which I assume is the time required to make a mask?) set by electron beam scan time or something?
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by vineyardmike on Tuesday August 25, 2026 @04:35PM (#49440234)
How much of that 16mo is design versus just production? If there was a “plug and play” chip where you just BYO weights, how long would it take?

The bigger issue seems to be that these chips can’t hold that many weights at the moment.

(I’m curious if chips with large weights in them would be more tolerant or less to yield issues. If you flip a few bits in the weights, does it really matter at scale?)

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by RealityVoid on Tuesday August 25, 2026 @06:13PM (#49441409)
Talaas, from what I understand is building stuff just like that. The infra is the same and the weights layer is all you need to change. I guess you could half etch the chips and then finish them with the weights only. I think their turnaround is 6-8 Weeks. The size of the models fitting on the chips at the moment is llama 3 I think?
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by derefr on Tuesday August 25, 2026 @06:43PM (#49441693)
> I guess you could half etch the chips and then finish them with the weights only.

Basically a https://en.wikipedia.org/wiki/Gate_array. (The non-field-programmable kind.)

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by jeremyjh on Tuesday August 25, 2026 @05:34PM (#49440987)
I think Sol is already good enough though.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by basilgohar on Tuesday August 25, 2026 @08:52PM (#49442837)
"640k (token context) should be enough for anyone."
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by jerf on Tuesday August 25, 2026 @09:10PM (#49442952)
I know what you're saying, but modulo things like losing track of what year it is as time passes by, a current frontier model is going to continue to be useful for many tasks for many years, even moreso if it's 5-10x faster due to the chip architecture.

It's not that it would be the best forever, it's that it would be useful for plenty long enough to be worthwhile, even if there was better stuff available. In exactly the same way that this computer I'm typing this message on is not the latest and hottest cutting edge stuff. A 7 year old CPU, 7 year old Intel integrated graphics, an older NVMe disk, a mere 32GB of RAM... ok, that's one spec that's still pretty modern although it is slower RAM... but it's still plenty fast enough to comment on HN, even these seven years after it was cutting edge.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by dgently7 on Tuesday August 25, 2026 @11:21PM (#49443852)
exactly, but the "goes out of date" is bad when we talk about software.. but this isnt software, its hardware.

the youd have to buy a new one to get a better model is a FEATURE not a bug.

like if im apple... and i can put a sol level llm in an iphone, market it as privacy first you own your data personal assistant, integrate it all over the os... and then when there is a better model/siri make all the users buy a new phone... thats how they "win" ai.

the old standbys of better screens thinner cameras and batteries arent enough anymore. its basically tapped out. all modern phones are as thin as they need as big as they need as fast as they need and last all day on a battery...

apple needs a new number to up thing that people can actually feel/see. model generations could be it... every year faster, smarter, more capbilities and integrations.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by throwuxiytayq on Tuesday August 25, 2026 @10:23PM (#49443429)
> but it's still plenty fast enough to comment on HN, even these seven years after it was cutting edge

While it’s still too early to tell, I don’t think that’s how intelligence scales. Better models get you better solutions even to trivial problems. The ceiling for getting it done better is very high even if you’re not doing anything complicated. And difficulty isn’t uniformly distributed anyway - it seems to me that “mostly simple” tasks often have annoying 1% tails that low-intelligence models struggle with. I think we’ll see people chasing the top models for quite a while, or indefinitely - depending on the cost curve.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by Marha01 on Wednesday August 26, 2026 @05:09AM (#49445973)
Well, with proper compaction algorithm (like in Codex), it could indeed be useful for majority of tasks even in the future. The context length in Codex is just 272k, but it can reason well about much larger codebases due to good exploration and compaction algorithm.

I have yet to saturate the 1M context of Gemini, for example.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by mcny on Wednesday August 26, 2026 @04:26AM (#49445641)
I don't think this is true at all. I use opus 5 to build a small project. It bothers me that people simply handwave away concerns so easily.

The context size is still not nearly as big as it needs to be to store all the code an enterprise needs. And apparently as you increase the context size, there are more defects. So none of this is a solved problem. We are still in early days and there is a lot to be done.

I'm sure there are marketing people who will say "coding is solved" and other such snake oil but none of this is done, far from it!

That being said there is still enormous value in older models especially with tool calling which will let them access the latest data. I feel like we need to be a little more careful and the tools should cache in a smart way to avoid rework but clearly if we could have opus 4.8 level of work for like a one time payment of a system for local LLM it will have value for years into the future.

So I agree in a weird way that sol is good enough for certain tasks but really there is a long road ahead.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by kurthr on Tuesday August 25, 2026 @06:13PM (#49441404)
The metal masked ROM is basically only 2 metal/contact layers. It's not a full new design and tapeout. You could roll a new set of parameters every ~2-3months. It's not an architectural change. See statements below.

https://www.eetimes.com/taalas-specializes-to-extremes-for-e...

https://www.turingpost.com/p/taalas

https://cambrian-ai.com/taalas-launches-hardcore-chip-with-i...

Part of the key is that by moving even from 6nm to 3-4nm one could embed a 20-30B model as part of a MoE (or only a subset of activated layers) on a single reticle die (note B300s are already multi-reticle), with a separate predictive/dispatch model controlling them each on a separate chip. This is without even stacking CiM ROM die. Moving the layer activations (and KV cache etc) between die requires relatively high speeds (and low latency), but distributed with multiple die in parallel might well be doable even with standard multilane PCIe. Of course KV cache prefill could also be handled by external GPUs. I'm sure AMD will make some reasonable choices.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by MBCook on Tuesday August 25, 2026 @09:59PM (#49443263)
But that means your different chips all have different sets of weights and are different generations.

If none of that is baked into the chip as now then all the chips are running the latest weights every time.

Even if you could ignore the stuff built into the chip when the time came, at that point you just wasted money on silicon that’s useless in 2-3 months.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by geysersam on Wednesday August 26, 2026 @01:37AM (#49444428)
Why would it be useless in 3 months?
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by imtringued on Wednesday August 26, 2026 @02:18AM (#49444741)
Because on hacker news the only thing that matters is being in the current news cycle and not whether your business is profitable.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by Certhas on Wednesday August 26, 2026 @01:50AM (#49444524)
Imagine Anthropic gives you Opus of 6 months ago but at much higher speeds and much lower cost (that they might or might not pass on).

Would you use it?

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by tyre on Wednesday August 26, 2026 @03:59AM (#49445456)
Yes, this would basically obviate Sonnet and Haiku. If you consider them 1 and 2 generations behind, respectively (that's not really what they are), you can still get a ton out of those older chips. Not to mention people still use older Opus versions happily. (In part because they don't like the new Opus but still, the cost effectiveness is a huge boon.)
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by user43928 on Wednesday August 26, 2026 @06:16AM (#49446508)
GPT 5.6 Luna is already rather fast at 300 tokens per second, performs better than Opus 4.6 from what I know, and is very cheap.

I don't know why anyone would use Haiku.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by kurthr on Wednesday August 26, 2026 @01:51AM (#49444529)
Since the current NVL72 are still at the ~15%/yr failure rate it's not clear your new data center is going to have half it's compute in 3 years. If you're still running H100s they draw >10x kWh/Mtoken as new designs. All of these systems become dated, but not all of them require entirely new infrastructure.

If a ROM rack running a near frontier agent model at >10ktoken/sec costs <$1M (rather than $4-8M for NVL72s) and draws only 10-20kW (rather than 100-200kW), and doesn't require a completely new cooling and power system every time you update? There'll be lots of demand for GLM5.3 in a year.

What these don't do is TRAINING, they only do INFERENCE, but they could do it pretty well.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by MBCook on Wednesday August 26, 2026 @10:21AM (#49449747)
They fail that often? Damn.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by magicalist on Wednesday August 26, 2026 @12:15PM (#49451632)
> Since the current NVL72 are still at the ~15%/yr failure rate it's not clear your new data center is going to have half it's compute in 3 years

This would be a stupidly bad failure rate, basically the worst business decision you could make, especially if you're somehow on the hook for eating those losses (which seems to be the implication?). Is there a linkable source on this?

The only thing I could find is SemiAnalysis claims that 15% of Blackwells end up RMAed[1]. That appears to be a total failure rate, though, and if you're RMAing them, you're getting replacements. So that appears to be a pretty different state of things.

[1] https://www.dwarkesh.com/p/dylan-patel#:~:text=GPUs%20are%20...

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by kurthr on Wednesday August 26, 2026 @01:37PM (#49452931)
I'll note CoreWeave didn't commission the very first Blackwells until ~Jan 2025 so it hasn't been very long for RMAs. Furthermore, for training even when the NVL backplane can use the remaining GPUs a ~20% drop in performance means it isn't in the training cluster.

I've personally heard this from several sources in the data centers (installers, training, network). It's not uncommon for 10% of racks to fail on delivery. I hear that's improved somewhat from GB200 to GB300, but the number of FW updates from the time they ship, until they're commissioned is >>10. If an HBM or GPU or backplane supply/cooling fails, it is basically not swappable or repairable. You have a "dead" rack, and deliveries are on allocation so you don't get a replacement for months (eg some "RMAs" for early delivered parts in late 2025 are still dead racks 9 months later). "Tray" swaps are technically possible, but still quite rare, perhaps because debugging takes as much time as commissioning a new rack.

I don't want to out anyone, but these are similar comments:

https://www.linkedin.com/posts/neelmaster1_aiinfrastructure-...

https://www.hostzealot.com/blog/news/nvidia-gb200-nvl72-is-n...

https://introl.com/blog/gb200-nvl72-deployment-72-gpu-liquid...

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by WithinReason on Wednesday August 26, 2026 @02:22AM (#49444773)
Doesn't matter if the chip is 100x-1000x more efficient and faster, and you can just make a new one for new weights. Imagine being able to run GPT Sol at 1k tokens/second a year from now, at a 100x lower cost per token than now. Would that be useful? Or Qwen 3.8 27B at 10k tokens/second. The super long thinking that makes qwen so effective would take a couple of seconds.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by ben_w on Wednesday August 26, 2026 @08:37AM (#49447966)
> Imagine being able to run GPT Sol at 1k tokens/second a year from now, at a 100x lower cost per token than now. Would that be useful?

Given the current rate of change, it would be hard to guess either way. By some measures the cost at fixed quality score goes down vastly faster than that:

  A similar trend is evident in the cost of models scoring above 50% on GPQA, a substantially more challenging benchmark than MMLU. There, inference costs declined from $15 per million tokens in May 2024 to $0.12 per million tokens by December 2024 (Phi 4).
- https://hai.stanford.edu/assets/files/hai_ai-index-report-20...

15/0.12 -> factor of 125 cost reduction in 7 months.

But that may well be an extreme case. To show how broad the range is, another quote from the same publication:

  Depending on the task, LLM inference prices have fallen anywhere from 9 to 900 times per year.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by WithinReason on Wednesday August 26, 2026 @09:15AM (#49448586)
This is from 2025, how about the last 6 months?
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by ben_w on Wednesday August 26, 2026 @09:48AM (#49449138)
You tell me. Most of these reports take that long to get published, or even longer. Sometimes I even see new-ish reports talking about 4o.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by jurgenburgen on Thursday August 27, 2026 @07:48AM (#49463251)
I’ll admit that it would be an interesting product, assuming you could buy the chips / cards and slot it in commodity hardware.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by CrimsonRain on Wednesday August 26, 2026 @07:37AM (#49447238)
A model is not useless if it is not sota. Price, and speed are also important.

A 6 months old model that can run at 1/10th hardware and much faster too, can be much more capable than a sota model when you don't have unlimited budget.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by thoughtbefore on Tuesday August 25, 2026 @09:00PM (#49442896)
It may not matter. Think about why SOTA model companies are exploring chips. What do chips offer?

If SOTA models haven’t peaked, then the SOTA model companies would still be churning out better and better intelligence.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by calebkaiser on Tuesday August 25, 2026 @10:09PM (#49443330)
Google rolled out TPUs in 2015. AWS released Inferentia and Trainium chips in 2020.

If companies working on ML-specific chips was evidence that large transformer models have fully saturated their potential, the field would have been done circa GPT-2.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by edgyquant on Wednesday August 26, 2026 @09:45AM (#49449085)
Neither of those companies core business model was serving llms
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by calebkaiser on Wednesday August 26, 2026 @12:21PM (#49451708)
What? Both of those companies absolutely serve LLMs, and both of them would love for serving LLMs to be an even bigger part of their business. Not only that, AWS is Anthropic's primary compute partner for training and inference. They literally use the newest generation of the Trainium chips I mentioned before: https://www.anthropic.com/news/anthropic-amazon-compute

Chips are another axis for improvements in training and inference. Orgs large enough to explore the space have been doing it for at least a decade now. This is just a silly line of reasoning based on the faulty assumption that somehow, looking for increases in efficiency in training/inference means teams have reached some theoretical limit in model capability.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by edgyquant on Thursday August 27, 2026 @11:04AM (#49466047)
Notice how I used the words “core business” but that they didn’t do business at all
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by tintor on Tuesday August 25, 2026 @10:41PM (#49443577)
They could etch the model architecture, without the weights into the chip.

This way newly post-trained model can be loaded and served the same day.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by nerdsniper on Tuesday August 25, 2026 @11:57PM (#49444059)
> Maybe once LLM improvements asymptote further?

Maybe! But it also doesn't require the rate of improvement to slow down. As long as some current model is eventually "good enough" for general use, it could still be a market-killer at a very low marginal price thanks to ASIC. Even if slower, much more expensive models are 10x better, that doesn't actually diminish the utility of the ASIC model, as long as it's "good enough".

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by lelanthran on Wednesday August 26, 2026 @02:07AM (#49444666)
> Note the ~16 months that the article quotes as being insanely fast to get this chip to tape-out (read: start producing).

Yeah, with any luck it would put pressure on Nvidia to charge less, and not just to OpenAI. With a little more luck, we would see all the other players do the same thing, driving down the price of actual GPUs from GPU manufacturers.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by thesz on Wednesday August 26, 2026 @10:26AM (#49449842)
I made some analysis half a year ago: https://news.ycombinator.com/item?id=47109252

It appears that to have working ASIC with the LLM baked into it we need to place and route macroblocks, and not a great variety of them. These macroblocks can be pre-placed-and-routed, available as masks already and shared between different LLMs.

Thus it appears that the tapeout delay can be substantially lower than a year.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by sebzim4500 on Tuesday August 25, 2026 @03:48PM (#49439620)
My guess is we only see this once they start saturating computer use benchmarks. That's a use case which would be extremely valuable at the right costs/speed, but the current models just aren't there yet.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by htrp on Tuesday August 25, 2026 @04:06PM (#49439870)
etched tried this.... it didn't go very well
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by anukin on Tuesday August 25, 2026 @06:55PM (#49441801)
I would assume asic based llm would work really well. Why did it not go well?
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by striking on Tuesday August 25, 2026 @07:44PM (#49442266)
https://chatjimmy.ai/ runs Llama 3.1-8B on an ASIC as a demo by https://taalas.com/ I believe.

That's quite a few parameters shy of today's trillion-weight behemoths, but it is fast.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by mchusma on Tuesday August 25, 2026 @10:33PM (#49443503)
You are correct. I think this is the bull case. It seems like this would be useful right now for some things (eg moderation).
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by fl0id on Tuesday August 25, 2026 @05:30PM (#49440937)
isn't that what they are doing with cerebras?
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by mkl on Tuesday August 25, 2026 @05:59PM (#49441276)
No, Cerebras holds the weights in SRAM - they are changeable, not baked in.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by andy_ppp on Tuesday August 25, 2026 @05:33PM (#49440963)
Yes, they could also sell me GPT Sol 5.6 or 5.7 on a chip and I’d probably buy it. It’s a really really useful model for me, I’m not sure how much better for coding I need it to be. For most things I find Sol good enough with a small amount of coaxing around my tastes.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by structural on Tuesday August 25, 2026 @06:11PM (#49441393)
Keep in mind that what previous work has done on a single chip with weights baked in was on a 8b parameter model. Sol is likely something in the 5T parameter range, perhaps higher. Serving the whole thing at BF16 is on the order of $3m in hardware just to serve it at all, and closer to $1-1.5m of hardware if it was being served as NVFP4. And power draw starting at high tens to low hundreds of kilowatts.

Let's say a magic set of chips comes along to host this. Maybe it's 2-3x more efficient in size and power. You're still talking a form factor that's a good chunk of a rack, draws tens of kilowatts, and could actually be sold at a similar if not higher price point because the OPEX is so much lower.

It may be useful but it's certainly uneconomic to spend >$1m to self host the model, plus ongoing power and maintenance costs, plus the cost to adapt whatever building you're in to be able to power it.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by dgently7 on Tuesday August 25, 2026 @11:30PM (#49443913)
ill give you that the way we talk about this ppl seem to think wed do this tomorrow, but in the 70s a kb of ram took an entire rack and tons of power also. Its seems equally plausible that we could go into a cycle of iterative refinement of baked model hardware that would end up in "personal ai" just like we got to personal computing.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by structural on Wednesday August 26, 2026 @01:05PM (#49452420)
Baked model hardware is not the next step in the chain here, in the next 2-3 years we might hopefully see some HBF (high bandwidth flash) hardware to try and get at same memory bandwidth today at a somewhat lower price point and much lower power dissipation.

Something like the next iteration of Cerebras hardware paired with HBM for KV cache + HBF for weights could be incredibly strong here and much more likely to see away to make into a product with some lifetime compared to "let's bake a old model into a very, very large number of custom chips, design all the interconnects from scratch, and pray". Maybe in 10-15 years once this all matures.

Now if that works out that means in '29/'30 we could easily see a run on NAND that's even worse than the current DRAM price issues, on top of the current increases. Fun times if that happens.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by Caracas288 on Tuesday August 25, 2026 @05:41PM (#49441064)
Man wouldn’t it be cool to be able to slot a massive ROM AI chip into the external AI drive of the pc…
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by xyzsparetimexyz on Tuesday August 25, 2026 @08:30PM (#49442663)
It'd just be pcie probably
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by pantelisk on Tuesday August 25, 2026 @09:07PM (#49442939)
It should look like a NES cartridge! That you have to blow on its end to clear any dust and it should do a satisfying click when it slots in.

Cooling might be an issue though...

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by porphyra on Tuesday August 25, 2026 @05:35PM (#49440994)
Also right now Sol 5.6 Max is super slow but if it were way faster on a chip (like Taalas' Llama 8b demo) then it would be an extreme value multiplier. But the model is so large that "baking it onto a chip" doesn't seem straightforward.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by redox99 on Tuesday August 25, 2026 @05:59PM (#49441267)
That'd be ungodly expensive.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by Aurornis on Tuesday August 25, 2026 @07:01PM (#49441863)
> I know this is what Taalas was doing (acquired by AMD), here was their demo, https://chatjimmy.ai/ which is based on Llama 3.1 8B. It feels like this should start to happen soon.

Taalas needed a giant chip (6nm) for an 8B model.

At best you could use a more advanced node to try to put a MoE model across several chips working together, but you can’t have GPT Sol size models on a single chip like that.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by greenknight on Tuesday August 25, 2026 @07:14PM (#49441987)
Nope. But we are hitting some pretty impressive levels with 128B models.

The other thing is, a lot of the time, model performance is improved with more 'thinking' time.

The thinking time is just more tokens... but instead of say 1000 tokens or 10,000 tokens worth of thinking its 1,000,000... how does that improve model performance? Could a 128B model hit levels of GPT Sol?

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by cherioo on Tuesday August 25, 2026 @07:30PM (#49442140)
Thinking generates a ton of tokens. These baked in chips tend to not have a lot of memory for context. I am not sure taalas supports Thinking at all.

The more problem like these they solve the more they will look like GPU.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by nextaccountic on Tuesday August 25, 2026 @07:43PM (#49442261)
couldn't one just add some hundreds of GB of HBM?
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by kimixa on Tuesday August 25, 2026 @08:52PM (#49442844)
Yeah, but then there's the size of KV cache needing to be read through that HBM interface for each token, putting a hard limit on the tok/s based on the memory bandwidth.

On some models a large context can be a notable proportion of the size of the weights themselves.

For example, qwen 3.8 27b uses ~64kb/token for the kv cache - so for a 256k token context that's ~16gb of the kv cache for a ~54gb model (assuming 2 bytes-per-param/f16 for both).

So if the current non-baked-in chip is already memory bandwidth bound, as is often the case for current hardware and models, and the "only KV cache in HBM" chip has the same total memory bandwidth, it can only ever be (54/16)=~3.4x faster for the baked in-silicon model.

EDIT: I guess actually (54+16)/16=~4.3x faster, as the current implementation would need to read that KV cache too :)

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by jubilanti on Tuesday August 25, 2026 @11:17PM (#49443817)
> Taalas needed a giant chip (6nm) for an 8B model.

You're phrasing it like it was kind of an inherent technical limitation with this kind of burning weights into silicon. Which is also not new, it goes back to the 1980s with fixed function digital signal processors and little linear regressions or hardware classifiers for industrial control systems, all are the same basic principle.

It's just usually not worth it to go super small process node, because most models people thought to turn into silicon were pretty small parameter sizes. We're talking 10-100 weight regression or at most 2-4k weight neural net, used in some instrument or factory equipment. You can do a decent MNIST OCR with a 4k weight neural net. For this, 180/130nm is fine.

Or you might think it's required with their special 4-bit as transistor thing (plausible). It's more that when you're experimenting and iterating, TSMC 6nm is their advertised path for rapid prototyping at cost for proof of concepts. And that's already in hot demand, while good luck if you're a startup trying to break in with 3/4nm as your first run.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by Aurornis on Wednesday August 26, 2026 @01:56AM (#49444564)
> You're phrasing it like it was kind of an inherent technical limitation with this kind of burning weights into silicon.

It is.

As I said, they could have shrunk it with a smaller process node, but that's not at all close to what would be required for a GPT Sol size model.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by mf_tomb on Tuesday August 25, 2026 @07:11PM (#49441957)
"Baking in" a model into a chip is a bad idea because chips take 2 years to tape out and then you're stuck doing inference on llama 3 in 2026 when fable/sol are available. Every accelerator is a tradeoff between flexibility and performance and GPUs are already pareto-optimal
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by twobitshifter on Tuesday August 25, 2026 @07:58PM (#49442417)
It depends when the good enough level hits. Pretty sure we are almost there for most common applications of AI.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by dgacmu on Tuesday August 25, 2026 @08:33PM (#49442685)
That's only half the problem. OpenAI is contractually obligated, if you will, to believe that models will continue improving at an impressive rate for the foreseeable future (otherwise their valuation makes no sense).

If you believe that, then you should expect to get Sol-level performance out of a Luna-cost model within six months or a year. If you have a system with the weights baked in, that means you're going to end up serving that Sol-class model several times more expensively than it will take someone who comes along a few months later. (such as what recently happened with DeepSeek's update.)

And under that assumption of continuing advancement, baking things in doesn't make sense in general - it's a play you'd make if you think things are slowing down a lot. Which may be right but it's not OpenAI or anthropic's play.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by adventured on Tuesday August 25, 2026 @09:03PM (#49442914)
Assume a $800 billion valuation. $100 billion ad network. $30 billion op income. 26x price to op income ratio. It's right there for them to grab, or someone else to grab.

Their valuation does make sense if you believe: 1) they can retain a massive user base and 2) a massive user base can be monetized. Future value is almost always pulled forward these days for high growth tech companies.

An LLM the size of Google search in users is even more valuable than Google search. The ad market for LLMs will be even larger than search was (no matter what HN prefers).

The monetization part is the easier part. Silicon Valley understands extraordinarily well how to build ad networks. If OpenAI maintain their gigantic user base, a $100 billion ad network is a given bolt-on. They'd have to screw that up in an epic way to not get there.

Facebook - Insta - WhatsApp is an absolute dogshit tandem with a gigantic user base. $228 billion in ad sales and still expanding 10% per year.

Google knows this is what's happening, that's why they don't care about chasing Anthropic very much. They're busy completely remaking how their core search business works.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by imtringued on Wednesday August 26, 2026 @02:32AM (#49444851)
If a strange and quirky architecture can increase their margins, they could start becoming so wildly profitable that they don't care about their investor driven valuation anymore.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by usef- on Tuesday August 25, 2026 @08:51PM (#49442828)
The whole point is that it's supposed to be more efficient. But models are also still getting absurdly more efficient every year, so you're likely nullifying much/most of the advantage. 18 months is a long time right now (and 18 is only time to tape out, not operational in data centers).

Even if the balance was net positive, you would also not be able to train them against new tools/harnesses or knowledge. How many years do you expect to keep using them?

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by adventured on Tuesday August 25, 2026 @08:55PM (#49442865)
The good enough level isn't ever arriving. We're in the first or second inning for LLMs. They will rapidly subdivide in complexity, they will not stagnate in the next decade.

Beyond the model, when would you freeze processor performance, such that it was good enough? Because that's exactly what freezing on Talaas is premised around.

The semiconductor technology will also continue to improve. You lose twice. Talaas is one of the dumbest ideas I've seen in semiconductors in decades.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by Gigachad on Tuesday August 25, 2026 @10:20PM (#49443402)
Good enough will hit when the tech stops advancing quickly. You could have a "good enough" model but in 2 years if the general purpose chip can run it just as fast, there is no point having the single purpose one.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by pantalaimon on Tuesday August 25, 2026 @08:30PM (#49442661)
Well we'll see those surplus chips being repurposed for toys then. Who wouldn't want a new Furby that can actually hold a conversation.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by vunderba on Tuesday August 25, 2026 @08:37PM (#49442723)
I've seen several attempts even on HN of the LLM meets Teddy Ruxpin (or more accurately AG Talking Bear) but most of them offloaded the AI to some off-site servers.

I’d like to think that most parents would be weary of handing their children what basically amounts to a tape recorder that siphons all the data off to a large corporation.

OTOH, a completely local one (LLM + VAD + Speech Rec) would be a fun little thing to build.

https://en.wikipedia.org/wiki/AG_Bear

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by guhcampos on Tuesday August 25, 2026 @09:58PM (#49443255)
There are other options. I worked for a startup called NVXL and we were programming DNNs into FPGAs using OpenCL, on custom boards we built to plug into NVME. It worked great, but it wasn't fast enough at the time to compete with Nvidia, or even Intel AVX512. Ultimately the company failed, but maybe some hybrid like that could work for LLMs? I haven't been up to date on how DNNs and LLMs look like under the hood these days, but there's got to be someone doing something similar.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by murderfs on Wednesday August 26, 2026 @04:04AM (#49445497)
Closer to 2 weeks, as long as the new model fits: your model bits are entirely in mask rom, so you can change them with a metal only ECO that only touches two metal layers. You probably don't even need to run any timing analysis, etc. since all of the changes are going to be isolated in a gigantic square that's isolated from everything else.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by pkphilip on Thursday August 27, 2026 @02:05AM (#49460446)
I think it would make sense to upload the weights like a firmware into the chip rather than baking the weights itself directly onto the chip. If there is an option of easily updating the firmware from time to time it would work well.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by andsoitis on Tuesday August 25, 2026 @09:24PM (#49443049)
> For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough.

but you trade updatability, which I don't think is worth it yet.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by mchusma on Tuesday August 25, 2026 @10:09PM (#49443334)
Maybe! (1) Would SOL level intelligence be useful 3 years from now? 5 years? (2) would dedicated chips be the most affordable way to run this model in 3-5 years?

I suspect the answer to both of these questions is yes right now, but I agree it’s borderline.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by andsoitis on Tuesday August 25, 2026 @10:16PM (#49443378)
3 years is an eternity.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by imtringued on Wednesday August 26, 2026 @02:27AM (#49444817)
LoRAs are a parameter efficient way to update models. The fine tuning stopgap only has to work long enough to extend the lifespan of a model until the next chip comes out.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by lqstuart on Tuesday August 25, 2026 @10:05PM (#49443307)
Eventually, someone is going to do this in Minecraft
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by bigfishrunning on Wednesday August 26, 2026 @09:21AM (#49448675)
Lol i'm surprised this hasn't happened already. Minecraft is a great platform for scripting as long as you think every single other platform is too fast and convenient.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by raincole on Tuesday August 25, 2026 @10:08PM (#49443329)
It won't happen until IPO. If they do it now it'd be signaling that AI isn't improving fast.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by freakynit on Tuesday August 25, 2026 @11:10PM (#49443781)
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by zeusk on Wednesday August 26, 2026 @01:08AM (#49444224)
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by sabhiram on Wednesday August 26, 2026 @01:48AM (#49444509)
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by la64710 on Wednesday August 26, 2026 @02:09AM (#49444683)
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by grackasthebig on Wednesday August 26, 2026 @01:21AM (#49444311)
Yeah this feels like Altman not knowing engineering well enough to realize where the focus should be.

Like he is optimizing to keep providing a vanilla token factory when weighted chips are coming and local models will supplement.

My head canon is savvy chip execs will be etching architecture his OpenAI pioneered into their flagship products while trying to minimize how much foothold he can get in hardware. Murica done offshored it. Not ours to control.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by miki123211 on Wednesday August 26, 2026 @03:14AM (#49445132)
I really want this for Whisper, particularly in some kind of a power-efficient, portable form factor.

I think people underestimate how much of a revolution having an always-on, privacy-preserving personal notetaker / secretary would be.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by mike_hearn on Wednesday August 26, 2026 @04:39AM (#49445729)
AI is bigger than just LLMs. The real place model-hardwired chips will find value is in robotics. There you need very local, low latency inferencing with relatively stable models to handle motor control and navigation tasks. Higher level reasoning can be delegated to LLMs and run asynchronously.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by walthamstow on Wednesday August 26, 2026 @05:13AM (#49446005)
I don't think any underestimates how powerful it would be. It's the social aspect that is difficult. I wouldn't want to be sat in a pub with you and your always-on notetaker.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by fooker on Wednesday August 26, 2026 @04:30AM (#49445668)
Computer Science history has taught us that this decision was almost always wrong.

Every time a company has spent resources doing this, a competitor innovated on the software and made the custom hardware irrelevant.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by aleph_minus_one on Wednesday August 26, 2026 @08:13AM (#49447628)
> Every time a company has spent resources doing this, a competitor innovated on the software and made the custom hardware irrelevant.

This often happened, though not always.

An important counterexample are 3D graphics cards, which basically put the OpenGL/DirectX fixed-function pipeline into silicon. It took a long time and many iterations to make the pipeline more programmable until the 3D graphics cards turned into modern GPUs.

Even today, GPUs live on as separate hardware in a computer instead of having become integrated into, say, the CPU. Intel's attempt to do something like this with the Larrabee project [1] was discontinued.

---

[1] https://en.wikipedia.org/w/index.php?title=Larrabee_(microar...

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by fooker on Wednesday August 26, 2026 @08:43AM (#49448064)
It didn't happen for the original fixed function graphics though, CPUs kept eating their lunch every year or so with fun rendering techniques. We still use some of these techniques for 2d graphics.

Only when they became more general with shaders, and then added support for GPGPU, did it truly take off.

I think this generality is the lesson here, not the fact that GPUs are not CPUs.