At the end of the day, when something looks like a duck, walks like a duck, quacks like a duck; it doesn't matter how much the nitty-gritty details prove the thing is absolutely not "actually" a duck to the average person.

Same logic: You write like a bot, you sound like a bot, you behave like a bot, hence, who cares if you are a bot or not? Let's ban you.

For this whole discussion, it actually really does matter if GPT-4 is able to reason or not. That is the essence of the discussion. It's not whether or not it looks like it can reason.

This is the reasoning that is causing problems with Tesla's "auto-pilot," with dangerous results.

My own car includes cameras and can keep me from disaster by applying brakes and correcting steering, but it makes no claims whatsoever to be self-driving.

A Tesla car unfortunately looks like self-driving, walks like self-driving, and quacks like self-driving, but it turns it does matter how much the nitty-gritty details prove the thing is absolutely not actually self-driving, in that encounters with things outside of its training model seem to result in collision or decapitations.

It is the trust people place in their cars that results in disaster when they reach the edge of what the cars can handle. They aren't ready to take over the controls and understand their current situation in a fraction of a second.

Placing too much trust in LLMs could have the same effect. I've run into several situations where I asked Chat-GPT technical questions and it confidently gave me detailed answers that clearly represented absolute failure. Fortunately, I don't trust it blindly, or at all, so I wasn't fooled.

I'd prefer an LLM more like my current car, that didn't try to do too much, but that's just not in the nature of how they operate, so instead we're all one emergency vehicle parked across the street away from a terrible accident.

And indeed, the Chinese Room (viewed as a whole system, not just the guy in it) is doing reasoning.

Sure, but Searle's interpretation of the Chinese Room Experiment was never very convincing to me. I think the room + the person inside it, as a system, do speak Chinese.

If it walks like a duck, quacks like a duck...