I'd assume the closed weight models are all working on shrinking their parameter counts anyways. They too benefit from smaller models. It'd be foolish for these SOTA labs to not be working at reducing parameter counts.

The incentive is there, but their money making niche is to solve problems you can't solve locally with a 30b, so they're unlikely to stray into territory owned by ultra cheap to run open weights models.

I would say even without rampocalypse there would still be the strong incentive to innovate at the edge and under more extreme constraints. The incentives are just even stronger now.

I'm looking forward to seeing what types of new things people create over the coming years once there is less obsession with massive unwieldy LLMs. I think the incentives are just too strong to ignore.

This is just temporary though, right?

With the benefit of LLMs already being proven, in a couple of years we will have vastly better hardware for inference I guess.

I feel like now hardware is stagnating a bit, because the software side has moved too fast for the hardware to catch up. Once we settle on some good, optimal software architecture for the models, dedicated hardware will easily increase throughout by 10x or 100x, for a fraction of the cost.

LLMs seems quite simple, maybe we'll be able to print/assemble at home our own chips with the desired models/weights.

Maybe we'll have model weights being shared like game cartridges.

I personally think of this like sorting algorithms. Quick sort does the same thing bubble sort does so why do we need quick sort? Pushing for efficiency drives innovation. It does this for many reasons but a big one is that putting a cap on a resource forces you to consider the others available and often you find that all it took was a little effort and suddenly the alternate path that looked a little worse is actually better than you realized.

This has a lot to do with how MCTS works BTW. The current best path is often only the current best path because a lot of investment has been sunk into it. If you were to put equal resources into a different path you may find that it was actually far better. It is just that the early rollouts favored the other 'best path' so you sunk a lot of resources into that one. We are very early in our exploration of LLM architecture. I highly doubt we are anywhere near the best path right now.

Definitely, LLMs are highly ineficient now.

The diffusion models are interesting, but those also seem hacky.

I think the next form of AIs will be simpler and more abstract.

The building blocks of our brain don't have the notion of a "token" embed into them, it's lower level that that.

I think first step is to find a better way to represent information.

LLMs shouldn't "compute" stuff using language tokens, but some other, more efficient logical mechanisms. LLMs should first "feel" the solution, reason internally in that optimised space, then, only when interacting with a human should it convert all that into actual tokens/language.

The problem with that is we don't have any kind of training data in that abstract sense, maybe we could use RL to figure that out but current RL techniques are too slow and prone to breakage that anybody trying to use them to train a big enough general model (LLM, diffusion, world model, etc) will either fail or have to make a very very big investment.

The other option is maybe hook up humans to EEG or the likes and map their brains while they solve different kinds of problem, or just see and feel the world around them

> This is just temporary though, right?

Well, everything—even human life on Earth—is just temporary, but RAM supply lagging centralized-AI-driven demand increases continuing to squeeze the consumer market may not be a short term phenomenon.

> LLMs seems quite simple, maybe we'll be able to print/assemble at home our own chips with the desired models/weights.

So, the solution to the RAM crunch is “everyone has their own home chip fab and deals with the raw material supply and hazardous waste disposal”?

I...don't imagine so.

Hazardous waste?

My idea was more like you get some pre-made chips, that you can maybe assemble together configure at home with your desired models.

Maybe each one of this chip is a layer, so you can stack as many layers as you want.

HBM modules that we can just attach to our GPU? Kinda like CPU RAM.

You're not thinking through the way economics works.

If, hypothetically, the thing you wish existed, why wouldn't it be valuable to high-utilization cloud providers?

And if useful to them, why would it be sold at a lower price than the (extremely high) price those cloud providers can afford to profitably pay?

The only solution to the hardware crunch that eases the cost/access barrier is if there's such a glut of excess supply of newly-commoditized inference hardware that the price craters.

And you've yet to propose a feasible manufacturing scenario through which that would be possible.

Well, before AI the RAM prices were cheapest ever, even if cloud providers were still in demand for hosting and cloud computing.

I hope the supply will increase at some point, I doubt cloud providers will be able to buy everything, especially if the competition will be high and the price of AI goes down, so there won't be an easy way to make profits by serving models.

> so there won't be an easy way to make profits by serving models

That's the only way I see things going, which ultimately comes down to the market losing faith in AI as a profit engine.

Which is probably going to take one of (a) OpenAI / Anthropic IPO failing or newly public financials looking bleak or (b) a high-enough profile large AI company failing (unfortunately most of the large enough to shock the market ones have diversified and durable cash generation engines).

Unfortunately?

So the solution to the chip supply problem is “pretend there is no chip supply problem and everyone gets the chips they need, but for some reason does some assembly task at home”?

If we could just do the first part and get reality to conform, the rest would be superfluous, but I don't think we can do that.

This is not a solution for now, just a moonshot solution for maybe 5-10 years from now.

There's no immediate fix for the RAM supply issue, but as long as it's manufacturing issue, not a raw materials issue, time always solves it.

Its not a solution ever, because it takes as its premise that the problem is solved and then goes on to suggest something else on top of it (the purpose of which is unclear.)

I fear the future of local models will be controlled by governments. I feel like some time soon there's going to be a crackdown on what is available to download, what is hostable, and what is "acceptable". I partially suspect it has something to do with why 128 GB seems to be the most you can currently purchase for a single machine, despite the price.

And we push ever closer to the edge of cyberpunk. Psst, interested in some black market heretical Mythos weights?

Let's not forget the Bitter Lesson. Small models sound really nice but at some point you're just fighting the laws of information theory. Efficiency gains on the small model side are nice, but efficiency gains + giant model tends to be even better...

I feel like the industry has quietly moved past the Bitter Lesson. In 2023 the story was naive parameter/data scaling and “emergent” intelligence properties. But there wasn’t enough data or compute to keep pushing in that direction, and the gains from it have been sublinear anyway.

Now, the labs spend enormous effort curating data pipelines to fit the models to a large assortment of very specific tools, tasks, harnesses, domains, etc. They also kind of fit to benchmarks by creating loads of synthetic training data that resembles benchmark tasks. None of this feels like the “scale up primitive methods and turn off your brain” message Sutton originally delivered.

How many bits of information are in a real brain?

You're totally right that we're far away from brain-level efficiency, but I'm just saying any efficiency gains we make towards small models will likely be felt on large ones as well, and we'll all move the goalposts to what the frontier can do. Hypothetically, getting GPT 5.6 performance in a 30B model would be amazing, but just imagine what you could do with a GPT 5.6-sized model at that point.

Don't know but a single neuron as a computation unit is far more complex than an ANN [1].

[1] https://christofkoch.com/biophysics-book/

More compute/larger datasets during training != larger models. The Bitter Lesson was that just scaling things up beats custom hand-crafted optimizations. Up until 2024, we thought that meant scaling up the parameter count, but then that started to plateau. After o1 was released, we thought it was about scaling up test-time compute. Now, seeing how 30b models can easily outperform 200b models from a few years ago, it seems like what we need to scale is RL, at least for agentic capabilities. It looks like 30b is already enough for good agentic capabilities. Larger models aren't considerably "smarter" (especially since they're mostly MoE anyway, with something like the same 30-50b active parameter range), they just know more (better world knowledge), which lets them make more informed decisions. Maybe we just need to scale up the retrieval layer.

I'm just saying that any larger model with all of our optimizations of today will always beat a smaller model with the same optimizations. Until the smaller models + optimizations are at AGI levels I don't think anyone will really care about what smaller models can do other than for routing / toolcalling

I am literally betting my company on this being true.

Me too. I think there are a few waves we can ride here. Let's collaborate?

What company? I am 100% focused on this as a concept in my own internal research.

workingmemory.ai!

It's a bad bet, historically.

I'm having an extremely hard time thinking of companies that have prospered due to software optimization. Most of them were swept away by hardware advances, instead.

The 1980's US car industry comes to mind. Nearly wiped out because they refused to make efficient vehicles. SpaceX is arguably showing how a rethink towards efficient can take over an entire industry. I am sure there are strong examples in software as well but they aren't coming to mind.

I think when successful, optimization really just means 'finally built right' and people forget the ridiculously inefficient ways before.

I would argue about 30% of Apple's success was just from not being slow annoying bloatware. I could never stomach it, but I know a lot of people who basically rage-quit Windows for the Apple ecosystem. I did the same, but for Linux.

Many of the current internet titans started by making things more efficient and accessible. Google for search, Facebook for connecting to people online, Microsoft for working with PCs at a reasonable price, Amazon for buying online as well as AWS. There are examples in other industries as well, Toyota is famous for it for example. There are probably counter examples but efficiency gains can be a huge deciding factor making companies successful.

You just listed a bunch of 0 to 1 companies, not 1 to 10 companies.

They weren’t quantitatively better than previous companies. They were qualitatively better.

I think finding significant efficiency gains with LLMs and the like may lead to qualitatively better products. Looking at people's experiences to DSV4F makes me believe that even more than before too.

I don't think people are realizing that speed can allow for categorically different user experiences that are more than just "worse than frontier capabilities but faster".

The headroom for hardware advances is a lot lower now than it has been for most of the industry's existence, when Moore's law held strong. Now we find ourselves limited by cost, physics, fab capacity, and complexity of spinning up more fab capacity.

Betting on innovation continuing to figure out ways to squeeze more out of less has historically been the right move. Look at Apple.

And I'd argue "hardware advances" are more proof of optimization.

IBM wants a word

On what? OS/2 didn't fail due to lacking software optimization.

I think the path of least resistance will end up being the cheapest and that is scaling up the parameters a ridiculous amount until you get highly capable models that can develop/distill/design the RAM efficient models. Going straight for low param is foolish and just a cope by smaller labs because they don't have the compute/talent to train the large ones.

This is 100% true for pretrains, likely true for RL as well although maybe there is some benefit to smaller activated params there. There is of course 0 benefit to small dense models relative to large sparse ones that are equally as memory efficient if you have enough computers.

Many on HN are in deep denial about this imo.

I honestly hope to see this across all applications, games, services, operating systems, etc. We've been in a period of wasteful RAM usage for over a decade. Constraints, whatever their origin, can be a good thing.

Same here, back to when algorithms and data structures mattered.

If China makes half decent RAM I would bet more on things like 128GM of RAM being the default on low spec laptops 10 years from now.

While I do love optimized software, the hardware side, especially for PCs, has been stagnating for way too long. At least now we have a valid use case for doubling available RAM every 2-3 years again.

I had a reasonably beefy Lenovo consumer line laptop that I bought in 2011, 8GBs of RAM. Its screen hinge broke and I couldn't repair it but I'm fairly sure it was otherwise still usable in 2023-24, once the HDD was replaced with an SSD. I think even now entry level laptops are sold with 8GB of RAM.

By comparison a PC from 2000 was utterly unusable in 2012-13.

I count on a 128GB baseline in 10 years. Beyond the current atmosphere of despair, I really want to see what Apple especially is cooking. Local AI is right up their alley and the current scarcity is unacceptable for them in so many ways.

I got the feeling laptops gonna feel very different in 2036.

Given local AI requirements, I expect minimum configurations for higher end machines to accelerate quickly. Yes, an entry level MacBook Air might be at 32 GB, but the entry MacBook Pro should quickly have 128 GB as a baseline. All this assumes we can actually make all the RAM we need. Until then, we’re going to be artificially capped.

Really? I feel like because nobody has RAM they're being pushed to the cloud frontier models. If we could all have our own 64GB+ VRAM GPUs, I feel like the open weight model scene would be even stronger.

Right now there is a stark difference between what smaller models can do and bigger models can do.

Smaller models are suitable for simple tasks like classification / summarisation while larger models are better in agentic capabilities.

Depends on the type of agentic task though. For simple operations, a small model can be quite beneficial.

I hope ssd streaming gets more popular, maybe more breakthroughs like that will help change things