RLHF is only tweaking the LLM to be more "well behaved". You wanted to say RLVR (reinforcement learning from verifable rewards). I know because I am building such RL gyms for computer use.

sure whatever, you get the point :)

i remember a year ago i'd have to tell the llm to use the bash tool to compile to see failures. these days, for every task it

1. throughly inspects git history

2. runs the full test suite to get a baseline

3a. for each change, runs (a subset if i'm lucky) the testsuite

3b. writes about 3 billion lines of tests, including tests to test tests

4a. uses the harness's screenshot tool to check each visual change

4b. for webapps, spends several minutes poking at it with some kind of browser devtool

5. runs the entire "smoke test" suite, which often takes several minutes and regularly interrupts my keyboard focus

6. runs the full test suite at the end

even small changes take at least 10+ minutes to complete, but with much higher accuracy than a year ago.

to me these things are dumb but persistent, useful but not intelligent.

> to me these things are dumb but persistent, useful but not intelligent.

Then you have a very uncommon definition of the term "intelligent", because that's roughly what I would expect an intelligent, conscientious developer to do. Many, many bugs stem from developers who believe themselves to be too "intelligent" to need an extensive test suite.

i would expect someone who can write good code in a single pass to be more intelligent than someone who requires multiple tries.

same with anything else really. a better chef can nail a dish in one or two attempts, whereas a worse but persistent chef might take 100 attempts to achieve the same quality.

or likewise, i'm not as good as a basketball player if it takes me 100 attempts to nail a three pointer when an nba player does it 35% of the time - despite both of us eventually achieving the goal.

I’m with you on this. LLM coding agents are incredibly fast, persistent and knowledgeable, and those are all really useful. But they still make a lot of stupid decisions. Without a feedback loop they just go wild. They write tests, which is a form of feedback, but without direction they just grow vast jungles of pointless tests, while the structure of the code itself gets more and more tangled.

Maybe the LLM is smarter than you give it credit for. Maybe the LLM is intellectually humble and self-aware enough to know that even a small code change can easily break something it hadn't thought of.

> RLHF is only tweaking the LLM to be more "well behaved"

Not quite right - RLHF is about aligning with human preferences (because it trains, as the name implies, on signal from humans). This does not necessarily correlate with behavior. We use RLHF across any number of domains and with any number of goals, not just alignment.

RLVR is a (more recent, better in some cases) way to automate the training loop without the human being involved, because you give the model a way to verify its own results and train on that as a signal.