I disagree. A lot of people skip disclaimers all the time, and when it comes to things like this, I think engineers of these systems have a duty to consider the possible consequences.
> Doctors give objectively bad advice in an authoritative manner all the time.
That doesn't mean we ought to automate the process.
> ChatGPT is a great way to get a second opinion, and at least could be really nice for truly rare conditions that doctors have a hard time diagnosing.
I don't think it's a great way to get a second opinion, and I think it's especially a bad idea to try to use it for diagnosing rare conditions. By the statistical nature of the language model, the more rare the condition, the less likely the model will predict a string of words that accurately diagnoses it.
> Also, don't confuse "biases in the training data" with "facts I don't like." It seems people often do.
The problem is not the actual facts but the inferences drawn from those facts when they are used to statistically generate responses by a language model.
I just asked ChatGPT what should be tested/what could be the cause of the fatigue and shrinking feet. It suggested a testosterone test (along with a few others), and that was indeed the reason. It also suggested some relevant specialists that might have figured it out.
I didn't find out until nearly 10 years after that appointment when I decided to really push for some more testing, after doing far more research than just typing it in to ChatGPT. I had to structure my whole life around my fatigue and didn't achieve what I could have. I also lost two inches of height as a teenager, in addition to my feet shrinking and my hands staying small.
I wish I had ChatGPT then, and it's surely helping people now.
Also, don't confuse "biases in the training data" with "facts I don't like." It seems people often do.