News for hackers. Stuff that matters.
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by Aurornis on Tuesday August 25, 2026 @07:01PM (#49441863)
> I know this is what Taalas was doing (acquired by AMD), here was their demo, https://chatjimmy.ai/ which is based on Llama 3.1 8B. It feels like this should start to happen soon.

Taalas needed a giant chip (6nm) for an 8B model.

At best you could use a more advanced node to try to put a MoE model across several chips working together, but you can’t have GPT Sol size models on a single chip like that.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by greenknight on Tuesday August 25, 2026 @07:14PM (#49441987)
Nope. But we are hitting some pretty impressive levels with 128B models.

The other thing is, a lot of the time, model performance is improved with more 'thinking' time.

The thinking time is just more tokens... but instead of say 1000 tokens or 10,000 tokens worth of thinking its 1,000,000... how does that improve model performance? Could a 128B model hit levels of GPT Sol?

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by cherioo on Tuesday August 25, 2026 @07:30PM (#49442140)
Thinking generates a ton of tokens. These baked in chips tend to not have a lot of memory for context. I am not sure taalas supports Thinking at all.

The more problem like these they solve the more they will look like GPU.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by nextaccountic on Tuesday August 25, 2026 @07:43PM (#49442261)
couldn't one just add some hundreds of GB of HBM?
Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by kimixa on Tuesday August 25, 2026 @08:52PM (#49442844)
Yeah, but then there's the size of KV cache needing to be read through that HBM interface for each token, putting a hard limit on the tok/s based on the memory bandwidth.

On some models a large context can be a notable proportion of the size of the weights themselves.

For example, qwen 3.8 27b uses ~64kb/token for the kv cache - so for a 256k token context that's ~16gb of the kv cache for a ~54gb model (assuming 2 bytes-per-param/f16 for both).

So if the current non-baked-in chip is already memory bandwidth bound, as is often the case for current hardware and models, and the "only KV cache in HBM" chip has the same total memory bandwidth, it can only ever be (54/16)=~3.4x faster for the baked in-silicon model.

EDIT: I guess actually (54+16)/16=~4.3x faster, as the current implementation would need to read that KV cache too :)

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by jubilanti on Tuesday August 25, 2026 @11:17PM (#49443817)
> Taalas needed a giant chip (6nm) for an 8B model.

You're phrasing it like it was kind of an inherent technical limitation with this kind of burning weights into silicon. Which is also not new, it goes back to the 1980s with fixed function digital signal processors and little linear regressions or hardware classifiers for industrial control systems, all are the same basic principle.

It's just usually not worth it to go super small process node, because most models people thought to turn into silicon were pretty small parameter sizes. We're talking 10-100 weight regression or at most 2-4k weight neural net, used in some instrument or factory equipment. You can do a decent MNIST OCR with a 4k weight neural net. For this, 180/130nm is fine.

Or you might think it's required with their special 4-bit as transistor thing (plausible). It's more that when you're experimenting and iterating, TSMC 6nm is their advertised path for rapid prototyping at cost for proof of concepts. And that's already in hot demand, while good luck if you're a startup trying to break in with 3/4nm as your first run.

Re:OpenAI JalapeñO: Better Than Nvidia Blackwell by Aurornis on Wednesday August 26, 2026 @01:56AM (#49444564)
> You're phrasing it like it was kind of an inherent technical limitation with this kind of burning weights into silicon.

It is.

As I said, they could have shrunk it with a smaller process node, but that's not at all close to what would be required for a GPT Sol size model.