This is not necessarily true. One easy way to compare version N and N+1 is "how much work (test time compute) does N+1 need to do to match the previous capabilities of N". It is true that some domains are currently hard to verify (e.g. biology is more difficult than math), but this is less of an N vs N+1 issue and more of a data domain issue.

How is being faster equating to more intelligent . Speed can achieved in a lot of ways for lot of tasks, a simple lookup table can fasten most algorithms but that doesn’t make them “smarter”.

Most of the interesting tasks these days require a lot incremental work and tinkering. Basically you want to build that can spend a lot of time tinkering, and measure how long they can go while still producing useful results. You can kind of break it down into two axes:

1. How long can the model work and still make progress

2. How fast does it get to interesting places.

We see N -> N+1 models getting better at (1), but it does take work to measure because you need to find tasks that were impossible for N and possible for N+1.

However (2) is very easy to measure, because you can keep using the evaluations you used for the previous generation of models and see if the new system gets there more easily. This is useful by itself (cost savings) and also in practice generally translates into being able to solve harder problems than before, as you'd expect.

That's how many proper IQ tests are. They test less of the depth and the ability to solve complex problems, but more of the crunching power - how many different easy/medium problems you can solve in a fixed amount of time.

How would you be able to know that N+1 hasn’t scraped a source containing the questions asked of N1 and optimized its answers ahead of time? In many ways that would be the logical course of action when you’re tasked with creating an LLM that is assessed by criteria X/Y/Z.

This is not an intelligence problem. It is in part an operational problem (have you properly set up the test), but it is mostly an alignment problem. Is the model cheating or not. Alignment is, indeed, very difficult.

Which is my point. In order for self improving models to work they will need less human supervision. And that’s going to introduce uncertainty compounding on uncertainty.

It's also unclear how intelligence actually scales in practice. A greater intelligence would be better at researching intelligence, but also harder to improve, so why would it explode rather than go up steadily? What are the actual resource requirements of various types of intelligence like persuasion or embodied intelligence? Coding, translating, copywriting, could very well be low-hanging fruits, very easy tasks we're very bad at, which doesn't say much about the hard tasks we're good at (embodied/animal intelligence, most likely).

There is also the problem of alignment, which is verifying that version N+1 behaves in a way that version N approves of. This is ostensibly the problem that we humans have with respect to AI development -- how to make sure they align with our values -- but it's also true of AI with respect to its own successors. Improving oneself without losing the plot would necessarily be slower than throwing stuff at the wall like evolution does.

This is also why trying to do RSI is dangerous. We're likely to get systems that are better in ways that are legible and easily measured, and have much less influence over fuzzier and harder to verify properties like alignment.

Let them compete in the market.

Start a new company with version N+1 leading it, and give it some funding. If it outcompetes version N company, it can acquire it.

Market forces do not act fast enough for an “intelligence explosion”.

That is not a tight enough feedback loop

Maybe not in the real world...

Any tighter feedback loop will be reward hacked by stronger intelligence.

Ah, you mean VendingBench [1]

[1] https://andonlabs.com/evals/vending-bench-2

I wonder if there’s a market theory of the things being traded also becoming market participants over time

The winner is the first company to RELEASE THE HYPNODRONES

(Markets are a human construct, not a law of nature. There's no reason to think they'll reward intelligence).

I think in order to achieve AGI we have to simulate what life does. Imagine having a bunch of robots with the ability to adjust themselves, just hanging out without a goal, experiencing world on their own. Eventually evolution will filter out those robots that aren't good at, you know, "being intelligent in general" and break without self-repair, while the rest will grow into a society.

Biological life has reproduction and death because once given organism develops there's no coming back and you have to start again with new generation. But if robots know how to self-repair and self-change, then we can skip reproduction.

The difficult part is the scale. Evolution took a shitload of time while being unimaginably parallel in order to make human intelligence happen, while we want AGI fast and on a budget.

The core of my point is that human intelligence arose from two things - humans experiencing the outside world and trying to survive in it, and forming groups that communicate using language. We gave AI a language, now we need to give it a way to experience the world on their own.

First, there are plenty of auto-verifiable problems like ML benchmarks datasets, benchmarks on hardware / software performance for coding tasks, and lean certificates for math, e.g.

Second, I think RSI will not be used like N -> N+1 where N is the frontier, since doing those experiments at scale is too expensive. My guess is they are doing N -> m_{1,…,99} where m << N, and then taking the insights from that and trying it at successively larger scales.

> First, there are plenty of auto-verifiable problems like ML benchmarks datasets, benchmarks on hardware / software performance for coding tasks, and lean certificates for math, e.g.

Goodharts law? Benchmarks are measures.

Yes, and therefore RSI isn't RSI. You need to set up multiple environments to generate data sets, multiple RL envs and just train the models on different datasets, architectures and then compare the final results against each other. The RSI part is just generating these pipelines end to end automatically.

What the hell are you talking about? There’s like 600 benchmark tests you can do.

Picturing advanced AI models eventually resembling Philosophy majors arguing about the number of angels that can dance on the head of a pin while the rest of us humans are left wondering if they truly have achieved N+1.