It's a heck lot smarter than most humans I know

By what metric? Ability to write a regex does not appear on any IQ test I know of.

By the simple metric of being able to converse with them on various topics

Intelligence is manifested by the quality and timeliness of decisions. Talking has always been a faux demonstration of actual intelligence, for man or machine.

Fooled me!

Very well said!

The big problem is going to be some "not so intelligent" Human making decisions based on AI/ML output without verification.

That’s… not true. Action and speech aren’t at odds. They’re not even clearly delineated.

Where would basic planning fit here? Is it a “faux demonstration” (??) for engineers to talk through their ideas for approaching a problem before attempting it?

If IQ tests are your metric GPT-4 is already way above average.

It doesn’t reason, it isn’t intelligent. There are humans who avoid reasoning but the biological capacity for this is inbuilt.

An LLM is like asking Google to make a search results collage. It’s an entirely derivative essentializer