Agreed. Like lots of this AI speculation appears to require the belief that AI will figure out how to build supervised training environments without more data.

I think this is very unlikely, but world changing if true.

Additionally, what does super intelligence even mean and why would we expect it to be obtainable from the data that we have?

I'd be delighted if someone can provide some definitions for these terms.

> Definitely not true as you can't use reinforcement learning to increase that capability.

What? Citation needed.

Well don’t you need a bunch of humans to test your persuasion capabilities? How would you run improvement tests in a loop??

You’d launch something like ChatGPT and get millions of people to use it everyday.

What is the measurement? I don't think most people come back to a conversation and say "gee you really convinced me to do X, here's the proof that I did it" and even if they did that data would still be very messy and require verification.

Uh this seems exactly backwards - reinforcement learning with human feedback teaches only the power of persuasion. The only way to get rewarded is to persuade a human to reward you. Why do you think the models are all so damn sycophantic?

You can't put RLHF in an automated recursive improvement loop because you need humans for it.