> When you have AI models autonomously self-improve, you implicitly lose your understanding of how they work and accordingly your supposed control of them?

This was always the case with ML, even absent RSI. Big bag of weights is a pain to figure out exactly why it can do whatever it is that it does, leading to all kinds of blind spots about how they go wrong.

> Currently, LLMs are used practically exclusively by "professionals" who, more or less, know what they're doing.

Even the people who made them don't really know what they're doing, hence https://www.felonybench.com