I agree in general, but part of what makes ChatGPT great for coding is that I can test the answers. They either work or they don't.

> They either work or they don't.

Or they work for a selection of cases but not others. Or they seem to work in a limited number of tests but aren’t truly doing the right thing. Or they do work but have a security flaw that makes your service vulnerable.

Coding solutions have tradeoffs and subtleties which go beyond “it did the thing right a couple of times when I ran it”. If that’s all it took to verify code is good, there wouldn’t be any bugs; you can test code a person writes too.

Yes, ChatGPT can't do all the work for you. You have to do the work to test the code thoroughly and think of edge cases and possible tradeoffs and all those other subtleties.

But contrast this with other types of information. If I ask ChatGPT "how hot is the surface of the sun?" and it answers "15 million degrees Fahrenheit", I really have no way to tell if it's right or wrong. As a result, I really don't find ChatGPT useful for these sorts of questions.

(The example answer is wrong according to https://www.space.com/17137-how-hot-is-the-sun.html, albeit in such a way that I could see potentially confusing an LLM.)

It's not much worse than a random blog or a stackoverflow answer. They never give you a guarantee that their answer is right for you.

But the nice thing about ChatGPT is that you can give it more context than it is possible in a Google query, and you can ask follow-up questions.

Yeah, in my experience, Stack overflow answers suffer from subtle errors all the time too. Especially for basic questions -- the people answering and upvoting the answers are inexperienced devs, so they often upvote answers that are either the wrong way to do it (eg. reimplement something that has a very elegant built in way to do), or they just contain bugs (eg. because they mixed up the number of characters vs the number of bytes in a string).

They are still useful, you just have to make sure you understand what you are doing and read the relevant docs before you paste the code.

Actual code, yeah, ChatGPT probably will mess up.

Things like "How do I delete a value from a dict" probably either works or it doesn't.

Then again, for that level of simple stuff, one can probably search a PDF cheat sheet just with their eyes faster than they can type the question for ChatGPT

It might sound weird, but you can actually ask it to do this too.

On the surface it makes no sense to do this since it already gave you an answer. But that's not how LLMs work - they can simulate a variety of different "humans" depending on the prompt and presenting a different prompt persona (security researcher, code reviewer that focuses on edge cases, etc) yields a different quality of response.

Well, but you still don't know if any of those persona are correct. It's great that you can talk to it, but ultimately you have to make the final judgement call, as it will bullshit you, and it can be much better at bullshitting than most humans.

So basically the same as with an answer from Stackoverflow but without the user voting.

It’s not really an issue for me because I generally know very quickly if it’s wrong or hallucinating and can easily get it back on track.

And in general I find it’s more accurate and far more concise than anything that’s going to show up in a Google search.

I mean, there’s no way I’m taking what it says at face value or copy / pasting code from its response, but that goes for stack overflow, or any other source too. (maybe sans official docs)

I'm curious. How do you know it is speaking untruths? What kind of questions do you ask where you know what a wrong answer would be? Is this just like "remind me how to reverse a string in go" kind of thing? Where you know how to do it, but cannot quickly recall?

Yep, exactly. I ask things that I know the answer to, or at least know enough to understand what the right answer is going to look like. I just can’t recall the specifics or am busy thinking through something else. It’s basically just an assistant, saving me the time of digging through official docs, searching previous project code, my own notes, etc.

I mean, it works for politicians.

There’s no guarantee that the SEO websites answer isn’t hot garbage too. You’re rolling that dice either way