Depends what the topic is.

IME it hallucines rarely when asked practical programming questions but basically constantly when asked math questions beyond basic calculus.

The more abstract or nuanced something is, the less capable it seems to be unless you have a domain specific LLM trained on a corpus of quality data.

As a business owner with both applied experience and formal education on both the tech and the business sides, I wouldn’t trust random LLM-generated advice on business. There are nuances and a certain humanity in good business that an LLM is unlikely to pick up. And the Internet is filled with horrible advice and useless insight on running a business that it can’t distinguish from good advice and valuable insight.

My advice to anyone seriously considering a business is that a course or reading a good survey text on business will pay off in ways that ChatGPT can’t (yet?) give you.

It constantly invents functions that don't exist.

Deal with it like Wolfram, implement those functions! It's picking up on something. :D

It might be just a glitch but in the last few days it generates such blunders I suspect something must be going on. Like completely messing up random pieces of code on subsequent iteration. I don't remember anything like this before. Normally it would improve the code with each iteration (until going out of the context window and starting to "forget" things, but not putting random characters here and there on second reply).

Likely you are just moving slightly outside of its knowledge domain with your questions. Anything that is hard to Google will also typically generate bad responses, most coding problems are there but many programmers just work on the few common problems over and over.

> LLMs are only useful if you know the topic you’re asking questions about.

Not true, and you're missing a key point for the topic at hand. LLMs are useful if you know the topic OR if you have measurable outcomes. If you want to increase sales, and you ask an LLM how to increase sales, you will know if its ideas work if they result in increased sales.

Think of it like AlphaGo - it learned to play Go by playing with an AI despite having no knowledge of Go strategy, because it could determine what was effective based on outcomes.

But I would only know about the result for my sales in some weeks or months. I have to trust the LLM in the meantime - and your example implies that the LLM answer could be proven wrong. How is that different from asking a complete stranger about better sales strategies for my business? Or even throw a dice?

This seems like a good example of Gell-Mann amnesia as it applies to LLMs instead of journalism.

People are falling into the same trap of accepting authoritative sounding advice as true in an area they are not experts in.