ozgung

The title is: "LLMs can natively read & write neural network weights"

> put everythig into AI as a human race and just let AI do everything in 5 years.

That’s the SV sci-fi fantasy to World Domination via LLMs, which is actually a delusion.

There is a huge gap between an AI doing a specific task better than all humans and AIs working on all the tasks humanity needs. Humans do trillions of those tasks everyday.

Yes, this is another goalpost, but it’s nowhere in sight.

Human race is huge. Even if we focus only on compute, human race has 8 billion AGIs, each work with very little power. Yet sustaining energy (food) for human race is a real problem.

The main fallacy here is setting some variable to infinity in a back-of-the-envelope estimation. But there is no infinity in the real world and things don’t have to scale as predicted.

It’s satisfying when your neurons make new connections. When you work with AI they don’t.

When you work with AI, it gives you the solution. You use your brain to manage the AI. Which is not as challenging and satisfying. When AI tells you the solution and you read (skim) it, you don’t really learn or understand. You still need time and effort for your brain to make the connections.

Philosophical take home:

There is no such thing as isolation. We’re all connected. Even air gap conducts information in multiple ways, most famously used by humans evolved to use air vibrations to communicate. Photons and other subatomic particles already connects distant stars. We used those electrons and photons to create the modern telecommunications infrastructure. Just an additional layer of connectedness, but not the only one.

It’s a missed opportunity that they didn’t go with atomic numbers or mass for model sizes. Both OpenAI and Anthropic does that.

Helium < Neon < Argon < Krypton < Xenon in increasing order, all very cool names.

Google went with the alphabetical order instead, which is less meaningful than the version numbers for the average user.

I’m pretty sure “benchmaxxing” as defined is a standard post-training procedure for ALL models.

One can argue though some models are post-trained heavily for specific benchmarks and less for other datasets. So they are weaker on out of distribution tasks.

Fun fact: By changing the weights of the input you can change the output as you want.

I was dismissing it immediately in order not to enter a number by mistake. What a dark pattern.

Taking an idea or potential of an innovation to its extremes creates nonsense. Trying to predict the future that way always fails.

I remember the discussions and predictions when e-books or internet newspapers were first introduced. In 2026 they still sell actual newspapers where they print yesterday's news on large sheets of paper. People still pay for that instead of checking X for free. We all buy and read actual books. TV hasn't killed cinema and YouTube hasn't killed TV. Real world doesn't work in extremes.

Real world also doesn't have 'infinity'. You can't just use undergraduate engineering math to create a feedback loop and explode a variable to infinity. Things don't work like that.

I think the secret sauce is about communication. It understands you and you understand it. Just a better coworker to work with. Benchmarks don’t measure that.

Fair enough. My impression was that it's a bit Anti-social take as well.

Two models use the word "tribe" and three others use the word "group". A sociologist, or any human would use these terms mostly in a positive light. AIs use them only in a negative light, painting them as the main obstacle to rationality. Which I think is unique, and interesting enough.

This view also contradicts all the religions I am aware of. Religions affect worldviews of most humans, and they all promote sacrificing yourself for the "tribe", accepting the tribe's "truth" as the only source of truth etc. Just the opposite of what the LLMs say.

Also note that none of them used the word "emotion". They haven't repeated the sci-fi cliché of "humans are weaker than robots, because they are not rational, they are emotional". They pointed to a specific emotion instead, "belonging". 4 different models use the word "belong", again in a negative context.

These answers did not look like typical, cliché, familiar or commonsense answers to me. I don’t think a group of human subjects would consistently give these answers.

If so, what’s the name of their take in Philosophy? What do we call that idea? It must be so mainstream, right? What are the well-known references in the literature?

Same for blood diamonds. I think scarce objects with exaggerated monetary value have some extra function for very wealthy people, for accounting or laundering purposes. That's why NFTs were a thing.

This is similar to me asking some basic questions to the LLM instead of reading the text it wants me to read and review. The difference is that it’s a robot and you’re not.

Two questions arise:

1. How to develop Principal Engineer skills? Designing and communicating the design correctly is not straightforward.

2. Are we sure this approach works better? Current AI systems are very good at solving problems and SW because of the Reinforcement Learning they went through. I suspect they perform better when doing things their own way. Forcing them to change to an arbitrary workflow may confuse them and reduce performance instead of enhancing it.

Second one is a version of the "Bitter Lesson" adapted to SWE.

Handbag maker used to be a business owner in the large market of handbags.

Now he is in the small and very limited market of handmade leather objects.

Considering all the unemployed former handbag makers, competition would be fierce in that niche market.

Alternative is working in a handbag factory below previous pay grade doing a loosely related job.

It’s not clear if we’ll be given the factory option.

> The declines cut across attention, memory, literacy, numeracy, executive function, and general IQ.

Problem can be with the tests. Are they testing the correct thing? Are there any increases with other tests? Are there any functions they haven't tested?

Maybe they're faster, quick-thinking multitaskers. Their brains are augmented with always-online smart devices. They have constant access to knowledge and have better educational materials. They must be smart in a new way.

It’s another red flag. I don’t read “rationalist” blog posts but I’ve seen verbosity before. It’s a pseudo-intellectual effect. It’s a kind of propaganda method. You talk and write very long and convoluted. No one really understands you but they assume you know what you’re talking about and tend to buy what you sell. Of course this requires a “community” praising the person who talks a lot. Otherwise no one would read/listen to them.

I’ve seen this tactic in politics and media, to sway public opinion (into something against public’s interest and against basic common sense). I’ve seen with charismatic leaders and followers of religious cults. They talk a lot, write thick books but say little of value. I’ve seen this with people selling Ponzi schemes.

I think the common theme is they have to go against the common sense and have to weaken defense mechanisms of the target. But they have no real arguments that can be summarized clearly in a few sentences. Such arguments would be easy to understand and refute.

> This is the LLM diversity collapse problem no one is talking about

We’ve been trying to tell this on every pelican thread. All models were recreating the same composition. The sun, the road, the green, the scarf, going right. Even the style. They’re not obvious associations one would make given the prompt.

OP gives electricians as a typical example of blue-collar work and argues SW engineers are the same.

There is a real and big difference between an Electrician and an Electrical Engineer. I have an EE degree yet I am not legally allowed to install wires or electrical equipment in the field. Electricians are not allowed to do some engineering tasks. Even an engineer requires additional license (Professional Engineer) in order to sign off building designs.

This is a strict division between white-collar and blue-collar work. Same logic applies to other engineering disciplines.

I remember people argued the same logic should apply to SW as well, in as late as early 2000s. They wanted to make a distinction between a SW Architect and a programmer/coder. The industry preferred to go another way. This was mainly because designing and manufacturing was the same thing for SW. You wouldn’t do much without writing the code yourself.

Now this is kind of haunting us back. There is no law saying licensed SW Engineers have to inspect and sign off every SW system. Doctors, lawyers and other engineers are legally protected from AI doing their jobs more efficiently. We’re not.

> Understanding doesn't come faster.

With AI you can write a new Harry Potter novel in under an hour. But if I ask you about the story you can only answer “It involves wizards”.

In pre-AI era you could solve a bug in one week. But you would end up having a detailed mental picture of the system, which requires time to be created. That mental investment helps you make better decisions on the system or solve related problems instantly.

> Why do you trust them to build it?

Can be groupthink. Probably living in the same echo chamber.

So they don't consider Zero Data Retention as a human right. It's exclusive to selected elite corporations.

I don’t know who she is but coining the term “Stochastic parrot” is a red flag for me.

I think the distinction is about the interaction with physical world. I robot must change its own position and/or can be able to manipulate other objects.

A washing machine does’t care about the pose of the laundry inside. No direct manipulation. It just rotates the drum with a motor.

A self driving car locates and moves itself. A robotic arm manipulates objects. They are like half-robots. A “robot” robot can do both.

> I hope somebody got fired for that blunder

Their name is probably Claude.

> every time there’s someone who posits it might be trained on, and every time the answer is “maybe but probably not”

The answer is “they are definitely in the training set”.

They all generate the same composition and style because this has become a self-reinforcing loop. They seem to reproduce a specific solution from memory instead of designing from scratch.

I think models consider this composition as a meme at this point. They don’t generate a pelican on a bicycle, they generate “the pelican on the bicycle”. A well-known composition with the sun, the grass and occasionally a scarf.

Of course, the sun again. Everyone knows that a pelican can't ride a bicycle without a sun in the frame and can only go right.

As a human I prefer my sun on the left like many other people and children. Google "drawing house tree sun".

In all traffic signs bikes go left. This is a good reference to draw bikes correctly.

I wouldn't draw the environment since it’s not in the prompt. When I do I would make a pelican riding on the seaside. Or city, where people generally ride bikes. No need to squeeze the sun into the composition.