Honestly, I've found that the architecture described to and by LLMs is always a more rosy picture than what is actually generated, no matter how many times you do an adversarial review. It's less visceral than in image generation, but the pattern is the same -- the broad strokes seem fine but the details are awful.
The problem here is that the LLM hallucinates, so
* it will tell that something is a bad idea, even when it is not. * it will miss good ideas.
But here is the thing. Even discussing with a rubber duck can do wonders to your thought process. So may people who are noticing the usefulness of this procedure might be actually just doing a slightly better rubber ducking...
The LLM will often miss the most obvious simplifications. And if I ask it to present me with six approaches how we could solve a problem, chances are we will settle on number seven or eight, both my ideas after rejecting all the other six. But those first six were still valuable for coming up with the version we actually settle on. Making the rubber-duck talk is genuinely useful.
And the rubber duck is actually pretty good at the localized grunt work, so you can spend more time talking with it about big-picture stuff
Is that something recent? I’ve been doing this for years, even in high school where we were asked essays on contrast between different viewpoints.
I don’t need LLM to do this. It’s the 101 of any engineering process to not rush with the first thought you have and indeed try to explicitly explore the solution space and base your decisions according to defined tradeoffs.
But that does come with tradeoffs, and it's not the right thing for every project. But when it does work it does increase the amount of work you get out in the same time