thunderbird120
thunderbird120 on Hacker News
Lots of chiplet interconnects hurt your max performance pretty severely and drives up your complexity a lot. There's a reason they want to keep it monolithic.
I already harped on Cerebras, but their approach of just copy/pasting a whole bunch of identical functional blocks, over-provisioning the chip by ~8%, and then just fusing off blocks with defects allows them to effectively have 100% yield on a wafer-sized monolithic chip. This is very desirable, and just another reason I like their approach better.
It makes fairly little sense for consumer hardware, unless you're embedding an absolutely tiny model. Since you're embedding the entire model in the chip itself, required chip size scales directly with model size. The reason they have llama3.1-8B as their model in
https://chatjimmy.ai/ (their demo) is because it's very hard to do anything bigger than that, you literally start running out of physical space on the chip, and bigger chips cost money. You can't have a dishwasher using a chip the size of a B200 die to run a model you could run on a Raspberry Pi.
What's worse is that this is when it's already quantized to ~3 bits per parameter (which is fairly lobotomized). Yes, the chip will run it 1000X faster than the Raspberry Pi, but it will only be stupid faster.
Their press release explicitly lists that their HC1 puts the entire Llama 3.1 8B model on one 815 mm² TSMC N6 die, with about 53 billion transistors.
815 mm² is fucking huge. An RTX 5090 is 750 mm². A mid-to-high end consumer CPU die is something like 300 mm², with a lot of budget parts being significantly smaller than that, down to ~70 mm². Every square mm costs money.
If they upgraded from N6 to N3P they MIGHT be able to get as much as a ~35B class model on a a chip which fits in the reticle limit, maybe, probably not, because model weights aren't the only thing that needs to fit on the chip.
There are very serious issues with agentic performance in this setup, which is exactly where you would want something really fast. Their Llama 3.1 demo lists a context of 6,144, which is dramatically lower than the 131,072 Llama-3.1 supports.
Reasoning models are barely usable with contexts that short.
The reason for this is that to actually get those speeds, the KV cache needs to live in SRAM. You can't bake the KV cache into the circuitry since it... changes. They clearly don't have enough SRAM, and the problem gets worse the bigger you make the model since KV cache grows (sort of) with model dim. The longer you want to make your context, the more of your chip needs to be SRAM.
Frankly, I don't see the use-case for this tech. It's too expensive and too inflexible. Just doing what Cerebras did and making a wafer-sized chip which is mostly SRAM is a much better solution to serving LLMs at extreme speeds and you don't need to make a new chip every time a new model comes out.
This is a near perfect encapsulation of the exact type of person you should not take financial advice from.
Kimi Delta Attention (KDA), despite having "Attention" in the name, isn't really attention at all in any conventional sense. It's more like an RNN which can be efficiently parallelized during training. It's a very small modification to Gated DeltaNet, which can be described as an RNN whose hidden state acts like a small, editable attention memory.
Because it's RNN-like, it has an inherent idea that X comes before Y which comes before Z in the sequence XYZ. Transformers, by default, don't have that. They operate on sets, unordered collections of unique items. They have no idea where those items are in relation to each-other so you have to clue them in.
Because There are 3 KDA layers per attention layer, and 3 KDA layers before the first attention layer, every single token position is going to be able to learn information about where it is in the sequence before the first actual attention layer.
RoPE is actually a bit destructive, so being able to omit it like this is very convenient. Models like Gemma-4 have a similar structure with 5:1 Sliding Window Attention (SWA) layers for every global attention layer. These are cheap, shitty attention layers which handle local information and which go in-between the big powerful ones, KDA serves the same role in this model. In Gemma only the SWA layers have RoPE while the global attention layers omit it. SWA is actual attention, even if it only operates on a small sliding window, so it needs the positional embedding. KDA isn't, so it doesn't.
>Curious to see if it holds up at frontier scale.
I don't know how much more frontier scale you can get than this, but yes, there's no reason why that wouldn't work at larger scales. Honestly, more parameters just makes it easier for the KDA layers to communicate that positional information better.
Medical expenses would be a combination of cost of insurance / treatment (PPP relevant) and government transfers in a country with partially or fully public healthcare, which is why it's so important to do both.
I see. That paper is looking a median income from a PPP perspective but without accounting for taxes and transfers, which is a tiny bit unorthodox in this exact context which tries to get into the weeds slightly more than just standard PPP adjusted GDP per-capita or something like that. It's valid statistically, but typically if you're trying to adjust for cost of living it makes a lot of sense to use the amount of actual income the individuals have left over after paying taxes and receiving transfers from social programs. Otherwise you'll have a number which accounts for differences in prices but not differences in the proportion of "income" which is actually retained by the earner. This is what that wikipedia article I linked is reporting. Comparing two different countries with very different tax/social program policies with PPP adjustments but without tax/transfer adjustments is less than ideal.
It's not wrong, it's just not very useful. It's typically a more intermediary statistic rather than a final one when doing this type of comparison.
Yes, since the weights being updated are a small subset of the overall total it's manageable. Just like how each separate conversation currently requires you to store a separate KV cache, you'd need to store the fast weights separately. Both KV cache and fast weight content stores have to be conversation specific, so just setting a bit of extra RAM aside for "memory" isn't really a new ask, just a different format for an old problem.
The idea of periodically stopping to write blocks of recent context into a fast-weight state is interesting, but I think it liked it better when E2E-TTT[1] did it. It's a more flexible and elegant continuous learning approach.
Essentially it goes "You know how your model can remember its training data? Well, what if you treated its recent context like more training data and updated (some of) the weights using (mostly) the same process used to train it?"
The end result is very good at remembering things but also really good at adapting to new unseen distributions.
[1]https://arxiv.org/abs/2512.23675
>Its turn around time is ridiculous, it has to be maintained with specialized equipment/hangers, along with external contractor assistance.
>Compared to the Gripen, as an example, which can land on a freeway and be up in the air again in a few minutes.
I have no idea where people got the idea that the F-35 requires a major refit after each sortie or that it needs climate controlled hangars, but there's literally no truth to any of it.
The turnaround time for an F-35 after a mission in a wartime scenario isn't going to be much different from any other older fighter jet. Refuel, rearm, get back in the air.
One of the key requirements for the F-35 programs was to minimize extra care needed for the RAM (Radar Absorbent Material). Unlike older stealth aircraft the F-35's ram is "baked in" to the aircraft skin, rather than being a coating. The F-117 and B-2 require climate controlled hangars because their coatings are old and delicate, the F-22 doesn't, but needs regular touch-ups for its coating, the F-35 is just left sitting outside most of the time regardless of where it's operating, a desert, the arctic, a jungle, the deck of a ship, you just leave it out there. The only common maintenance done on the F-35's RAM is replacing a relatively small amount of special RAM tape which is usually used around the edges of the access panels which are opened for other types of maintenance.
Cerebras has effectively 100% yield on these chips. They have an internal structure made by just repeating the same small modular units over and over again. This means they can just fuse off the broken bits without affecting overall function. It's not like it is with a CPU.
That's what it's running on. It's optimized for very high throughput using Cerebras' hardware which is uniquely capable of running LLMs at very, very high speeds.
You prevent enemy air defenses from shooting down your aircraft by blowing them up as part of SEAD/DEAD missions, which is exactly what the US did.
Takes much longer to build, requires a much larger up-front investment, and requires a lot more land.
The footprint needed when trying to generate this much power from solar or wind necessitates large-scale land acquisition plus the transmission infrastructure to get all that power to the actual data center, since you won't usually have enough land directly adjacent to it. That plus all the battery infrastructure makes it a non-starter for projects where short timescales are key.
Precision bombing during WW2 was not possible at the required scale. To put a bomb precisely on target back then you needed something like a dive bomber, a tactic which is incompatible with strategic-scale bombing. Even "precise" methods using advanced analog computers like the Norden bombsight could only do so much.
>Under combat conditions the Norden did not achieve its expected precision, yielding an average CEP in 1943 of 1,200 feet (370 m)[1]
This means that 50% of bombs fell within 1,200 feet of the target, which is an absolutely awful accuracy if you're trying to hit anything specific.
This was further compounded during the campaign against Japan by the heavy reliance of Japanese wartime industry on cottage industries which were dispersed almost randomly within Japanese population centers, rather than being located within specialized industrial districts. From a purely strategic standpoint which is only concerned with destroying the enemy's ability to make war, the most effect way to disrupt these kinds of industry with 1945 technology was essentially to burn every building in the city to the ground. Other options were simply ineffective.
[1]https://en.wikipedia.org/wiki/Norden_bombsight
Nice to see that they added that, but that section wasn't in the article when I wrote that comment.
This article doesn't mention TPUs anywhere. I don't think it's obvious for people outside of google's ecosystem just how extraordinarily good the JAX + TPU ecosystem is. Google several structural advantages over other major players, but the largest one is that they roll their own compute solution which is actually very mature and competitive. TPUs are extremely good at both training and inference[1] especially at scale. Google's ability to tailor their mature hardware to exactly what they need gives them a massive leg up on competition. AI companies fundamentally have to answer the question "what can you do that no one else can?". Google's hardware advantage provides an actual answer to that question which can't be erased the next time someone drops a new model onto huggingface.
[1]https://blog.google/products/google-cloud/ironwood-tpu-age-o...
People would correctly identify that their standard of living is being reduced for ideological reasons without tangible individual benefits and would likely not respond well to that, resulting in a loss of political power for whatever movement instituted those policies and a reversal of said policies.
Yeah, that's the normal outcome for papers like this. Papers which claim to be groundbreaking improvements on Transformers universally aren't. Same story roughly once a month for the past 5 years.
If that were totally true you would expect to see more or less uniform ratios of HP/HD cells mixes across different product types, but that's very much not the case. Dennard scaling may be dying but it's not dead yet. You can still sacrifice efficiency to gain performance. It's not zero sum.
If it were that simple fabs wouldn't offer a standard cell libraries in both high performance and high density varieties. TSMC continues to provide both for their 2nm process. A tradeoff between power efficiency and raw performance continues to exist.
>Intel on 18A is literally TSMC's 3nm process + backside power delivery, which means more power efficiency, performance also less heat.
That's a pretty serious abuse of the word "literally" given that they have nothing in common except vague density figures which don't mean that much at this point.
Here's a line literally from the article
>Based on this analysis it is our belief that Intel 18A has the highest performance for a 2nm class process with TSMC in second place and Samsung in third place.
Given what we currently know about 18A, Intel's process appears to be less dense but with a higher emphasis on performance, which is in line with recent Intel history. Just looking at the density of a process won't tell you everything about it. If density were everything then Intel's 14nm++++ chips wouldn't have managed to remain competitive in raw performance for so many years against significantly denser processes. Chip makers have a bunch of parameters they have to balance when designing new nodes. This has only gotten more important as node shrinks have become more difficult. TSMC has always leaned more towards power efficiency, largely because their rise to dominance was driven by mobile focused chips. Intel's processes have always prioritized performance more as more of their products are plugged into the wall. Ideally, you want both but R&D resources are not unlimited.
The cost for both training and inference is vaguely quadratic while, for the vast majority of users, the marginal utility of additional context is sharply diminishing. For 99% of ChatGPT users something like 8192 tokens, or about 20 pages of context would be plenty. Companies have to balance the cost of training and serving models. Google did train an uber long context version of Gemini but since Gemini itself fundamentally was not better than GPT-4 or Claude this didn't really matter much, since so few people actually benefited from such a niche advantage it didn't really shift the playing field in their favor.
No, additional context does not cause exponential slowdowns and you absolutely can use FlashAttention tricks during training, I'm doing it right now. Transformers are not RNNs, they are not unrolled across timesteps, the backpropagation path for a 1,000,000 context LLM is not any longer than a 100 context LLM of the same size. The only thing which is larger is the self attention calculation which is quadratic wrt compute and linear wrt memory if you use FlashAttention or similar fused self attention calculations. These calculations can be further parallelized using tricks like ring attention to distribute very large attention calculations over many nodes. This is how google trained their 10M context version of Gemini.
Frankly, yes.
Many creative works these days require the effort and input of so many people, so much time, and so much money that they can't have a specific creative vision. Mediums like book, comics, indie movies, and very low budget indie games, where the the end product was created by the smallest number of people, have the most potential to be interesting and creative. They can take risks. This doesn't mean they will be good, most aren't, but it means that the range of quality is much broader, with some having a chance to shine in ways which big budget projects just can't. The issue with small teams and small budgets is that they are inherently limited in what they can create. Better tools allows smaller groups of people to make things that previously would have required an entire studio but without diluting the creative vision.
Will this also result in a tidal wave of low effort garbage? Of course it will. But that can be ignored.
The APU offerings from both AMD and Intel have been improving pretty rapidly recently but they're still pretty low end by dGPU standards. I can certainly see them causing the death of dGPUs in laptops but it's difficult to imagine a scenario where they're competitive with mid to high range dGPUs in desktops. I can't see either AMD or Intel trying to cram an iGPU that large onto one of their chips, it would be an extremely niche product and would be badly bandwidth starved.
The 18A naming convention is as simple as not wanting to put a decimal point in their node name. Simple as that. It's not like the the "nanometer" node names actually mean anything at this point anyways. Neither TSMC N3 or Intel 3 have any dimension in their transistors which are 3nm. I would be very unsurpassed to see other foundries copy the convention, it's a lot cleaner.
Intel 3 is an economically interesting node. Performance seems to be somewhere between TSMC N4P and N3B which is respectable but Intel 18A is due very soon after it debuts and incorporates GAA transistors and backside power delivery. It seems like Intel 3 is intended as an option for customers who need something very modern, but not absolutely cutting edge. I guess we'll see how much volume it actually gets over time.
The ISA is much less important than many people seem to think. The RISC vs CISC debate is beyond outdated at this point because no modern architecture actually works strictly like either under the hood. Organizations who did x86 architectures historically had much more emphasis on performance while organizations who did ARM had more emphasis on low power devices. The lingering engineering consequences of that history and the experience of the organizations doing design are orders of magnitude more relevant than the difference in actual ISA.