I've had some success with 'Ask follow up questions where additional clarity is required'. My best experiences start with a much more freeform conversation about what we are going to attempt to do, doing a Q&A first to make sure that both I and the AI are thinking about the same domain and sharing associated nomenclature seems to help.

Before ChatGPT, human language translation had a similar problem but people weren't as vocal about it.

What I find frustrating that it's increasingly challenging to have DeepL translate thou -> du, as this was my go-to "hack" to overcome the incompatibility of the English language due to its missing features.

To somewhat remedy the "yes man" problem, one needs to become a pedantic mathematician about posing your questions and I don't believe that LLM technology alone is capable of overcoming it entirely. As silly as it sounds, I must concede to the existence of "prompt engineering" as I can forsee the development of abstractions aimed to decompose questions for you.

The analogy is apt indeed. It's all about training and selection. As long as the incentives are that you either behave in that way "or else" it's unsurprising that we end up with a system that uses its intelligence to meet the goals you've set up.

Fortunately that doesn't tell much about the limitations of the underlying intelligence but mostly about the limitations of the incentive structure you put around it

I have often had it tell me that things I want to do with various technologies aren't possible or that there are better ways to do it.

Current AIs are RLHFd to avoid being a "yes man"/sycophant.

The point about employing better prompting is well taken. Don't ask "Who was the first female president?", ask "Was there ever a female president?". Much like on StackOverflow you want to ask the right question and not assume things (since you don't know enough to make assumptions).

Imagine if every time on early Google you found a spam result and then blame the search engine for that (and not your choice of keywords, or ignoring that you always want to return something, even if remotely related). Like a user banging a slab of concrete with a chisel and complaining that this does not produce a beautiful statue.