There are several reasons to forget:

  - copilots are useful
  - chat is entertaining and useful
  - future tech is coming
  - investment money

I don't know if I'm more forgiving of inaccuracies in a conversational interface, but I'm way less likely to notice them in the first place. Especially since the current crop of RLHF'd models are so eager to please that they say nearly everything with high confidence.

Yeah I think a lot of people think RLHF is a tool for increasing accuracy but it really is training it to be convincing.

Which should be rather obvious if you understand that it's basically a GAN. You have a discriminative model who's objective function is based on Justice Potter's description of porn: I know it when I see it. If you ask what types of errors might emerge from such a formulation I think it becomes rather obvious.

Which isn't to say that the tools aren't useful. But I have to add this fact because many people conflate any criticism with being dismissive of the technology. But technology is about progress, not completion. Gotta balance criticism and optimism. Optimism drives you and criticism directs you.

I think this is a more realistic notion of what AI danger is rather than the X-risk. It's our tendency to trust things given a certain means of presentation. There are certain things where error is okay, but many things where even a small error is huge (like the OP is mentioning). The danger is not so much in the tool itself but us using the tools in a lazy manner. It isn't unique to ML/AI, but ML/AI uniquely are better at masking the errors. It's why I dislike the AI hype.

Though I wonder how much of that is just that the format doesn’t encourage you to look closely enough at what you’re getting to see if it is right.

Personally having been a big fan of GPT-3, I was quite against ChatGPT because of this.

Completion models are obviously wrong very often. Instruct model was kinda ok, but you know it's a dumb machine.

Chat was a bit of an uncanny valley. I treated the instruct model like a child, but chat felt like having a conversation with someone of 80 IQ. It felt frustrating, and you ended up going "no no no, what I meant WAS ..." It felt like dealing with an incompetent colleague.

But I guess there's lots of views on it. Some expected it to be an oracle, even a god. Some treated it like Stack Overflow, then got frustrated that it was giving poor quality answers to poor quality questions. Some were just abusive to it. I suppose it's a mirror in a sense.

> I think that ChatGPT's success might be partly attributable to its chat interface. For whatever reason, a lot of people - including me! - are much more forgiving of inconsistencies, slip-ups, and inaccuracies when in a conversational format. Kind of like how you might forgive a real human for making a mistake in conversation.

The key term here is "conversation". If I query something from the machine and it disappears and rumbles and then prints off something like a 1980s mainframe, with paper that has those holes on the side that you tear off... and then it's wrong, it's wasted time.

Meanwhile with the conversation I'm watching it in real time, and can stop it, refine it, or ask or clarification immediately and effectively. There is an expectation of give and take and "talking through" things to get to an answer, which I find is effective. I don't need it to be 100% right all the time, just 80% and then start parsing answers out of it to refine it to 90% accuracy with high confidence.