> They either work or they don't.

Or they work for a selection of cases but not others. Or they seem to work in a limited number of tests but aren’t truly doing the right thing. Or they do work but have a security flaw that makes your service vulnerable.

Coding solutions have tradeoffs and subtleties which go beyond “it did the thing right a couple of times when I ran it”. If that’s all it took to verify code is good, there wouldn’t be any bugs; you can test code a person writes too.

Yes, ChatGPT can't do all the work for you. You have to do the work to test the code thoroughly and think of edge cases and possible tradeoffs and all those other subtleties.

But contrast this with other types of information. If I ask ChatGPT "how hot is the surface of the sun?" and it answers "15 million degrees Fahrenheit", I really have no way to tell if it's right or wrong. As a result, I really don't find ChatGPT useful for these sorts of questions.

(The example answer is wrong according to https://www.space.com/17137-how-hot-is-the-sun.html, albeit in such a way that I could see potentially confusing an LLM.)

It's not much worse than a random blog or a stackoverflow answer. They never give you a guarantee that their answer is right for you.

But the nice thing about ChatGPT is that you can give it more context than it is possible in a Google query, and you can ask follow-up questions.

Yeah, in my experience, Stack overflow answers suffer from subtle errors all the time too. Especially for basic questions -- the people answering and upvoting the answers are inexperienced devs, so they often upvote answers that are either the wrong way to do it (eg. reimplement something that has a very elegant built in way to do), or they just contain bugs (eg. because they mixed up the number of characters vs the number of bytes in a string).

They are still useful, you just have to make sure you understand what you are doing and read the relevant docs before you paste the code.

Actual code, yeah, ChatGPT probably will mess up.

Things like "How do I delete a value from a dict" probably either works or it doesn't.

Then again, for that level of simple stuff, one can probably search a PDF cheat sheet just with their eyes faster than they can type the question for ChatGPT

It might sound weird, but you can actually ask it to do this too.

On the surface it makes no sense to do this since it already gave you an answer. But that's not how LLMs work - they can simulate a variety of different "humans" depending on the prompt and presenting a different prompt persona (security researcher, code reviewer that focuses on edge cases, etc) yields a different quality of response.

Well, but you still don't know if any of those persona are correct. It's great that you can talk to it, but ultimately you have to make the final judgement call, as it will bullshit you, and it can be much better at bullshitting than most humans.

So basically the same as with an answer from Stackoverflow but without the user voting.