The problem with this is "how do you assure the AI isn't bold face lying to the defendant".

For a natural text generation/chat system like ChatGPT, at the moment this is effectively impossible and frankly I don't see it being sufficiently fixed for something as complex as teaching law to a non-law-student any time in the near future.

So you make this as understood as possible and the lawyer can fill in the gaps and clarify.

It's not like a "jailhouse lawyer" will do any better. Though it might cost something you didn't expect, such as a new witness against you or a shower pal.

I think these discussions always seem focused on the ideal state of the "AI is perfect and gives correct answers" and not the current state where the "AI constructs sentences that are structured correctly, but are filled with inaccuracies and illogical nonsense."

If we want to steelman AI why shouldn’t we also steelman the court system? Assuming a perfect AI is about the same as assuming a well functioning court system. Otherwise we should be assuming a flawed, potentially racist, confidently incorrect AI vs a flawed, excessively slow, definitely racist court system.

I don't see how it's the same.