That's why professional bodies have rigorous standards regarding liability. Every legal AI product has to have an attorney of record who is responsible for reviewing the output and ensuring it's conformant to requirements. (In other words, an AI looks a lot like legal process offshoring, where your contracts attorney might be working in Argentina, but there's a US attorney who's liable for the work product.) We've already seen one company try to worm an "AI litigator" into production without actual attorneys being in charge, and that was a clusterfuck well before they got within blast radius of an actual trial.

Likewise, a "medical AI" is going to be regulated as a Class II medical device, and trying to push it out into the market without FDA approval would be ... inadvisable. This means that we're a long way from an architecture for a LLM-based medical AI that's going to pass regulatory scrutiny, but that's a good thing.

Of course, having said that, there's nothing preventing someone from using ChatGPT to draft a pro se filing or trying to self-diagnose, but that's on them at that point. Using a general-purpose product to do something stupid that inevitably ends in a FAFO scenario is a time-honored American tradition.

> Likewise, a "medical AI" is going to be regulated as a Class II medical device

Doctors sometimes use Google to get information to help them make decisions, but I assume Googling isn't regulated as a Class II medical device. These days, Google doesn't even just return search results anymore, it also shows ads and tries to answer some questions by itself. Does that mean doctors are already using unregulated 'medical AI'?

No, they’re saying that you won’t be using ChatGPT as a medical device without a real doctor present.