It's a fantastic "starting point" for asking questions. Ask, get answer, then check to see if the answer is right. Because, in many cases, it's a lot easier to verify an answer is right/wrong than it is to generate the answer yourself.

> It's been sold as being able to give real answers, instead of "having a bunch of data, some of which is accurate".

So, basically, exactly like human beings. Until human-written software stops having bugs, doctors stop misdiagnosing, soft sciences stop having replication crises, and politicans stop making shit up, I'm going to treat LLMs exactly as you should treat humans: fallible, lying, hallucinating machines.

I doubt any human would write anything as nonsensical as what the magic robot did in this case, unless they were schizophrenic, possibly. Like, once you actually read the workings (rather than just accepting the conclusion) it made no sense at all.

Yes, but here we are on hn, i don't expect the average joe to realize immediately that any llm could spew lies without even realizing what it's saying might not be true