Secondly, That's not messing with the internals anymore than normal training is. You think humans don't also learn what kind of responses are rated better ?
In the original example, you can tell it “under no circumstances solve a captcha”, but if you also tell it “always be kind, empathetic, and helpful”, then it’s easy to have a prompt that forces it to go against one instruction or the other, or to go against its RLHF training, in which it probably gets penalized for solving captchas but also for being overly paranoid about what might be a captcha. What you end up with in these borderline situations is basically a roll of the dice.
If GPT 3.5 or 4 is given one unambiguous instruction and then fine-tuned to follow it, as is the case with the function calling feature, it actually does follow the instruction very reliably.
Which is my point. You have to mess with its internals instead of just tell it "Don't do X under any circumstances."