Hallucinations can be manifested in different ways. Using nonexistent APIs is just one of them. The LLM could just as well hallucinate code that doesn't fix a problem, or hallucinate that a problem exists in the first place, all while using existing APIs. This might not be a major issue for tasks like programming where humans can relatively easily verify the output, but in other scientific fields this can be much more labor-intensive and practically infeasible to do, as this recent example[1] showcases. So hallucination is a problem that involves any fabricated output that isn't grounded in reality.

Which isn't to say that it is a universal problem. In some applications such as image, video or audio generation, especially in entertainment industries, hallucinations can be desirable. They're partly what we identify as "creativity", and the results can be fun and interesting. But in applications where facts and reality matter, they're a big problem.

[1]: https://news.ycombinator.com/item?id=44174965

You can test every line of code in your program, but how many people actually do?

It's one thing if you are just creating a throwaway prototype, or something so simple that you will naturally exercise 100% of the code when testing it, but when you start building anything non-trivial it's easy to have many code paths/flows that are rarely executed or tested. Maybe you wrote unit tests for all the obvious corner cases, but did you consider the code correctness when conditions A, then B, then C ... occurs?). Even 100% code coverage (every line of code tested) isn't going to help you there.

> You can test every line of code in your program, but how many people actually do?

In my mind, that's what separates genuinely excellent professional programmers from everybody else.

I think it's perhaps more that you learn to write code that is easy to test and debug, consisting of some minimal set of simple orthogonal components, etc. You test every function of course, but learn to intuitively design out these combinatorial complexities that could nonetheless still be lurking, and pre-emptively include assertions to try to catch anything you may have overlooked.

This argument is the reason why LLM output failing to match reality was labelled 'hallucination'. It makes it seem like the LLM only makes mistakes in a neatly verifiable manner.

The 'jpeg of the internet' argument was more apt I think. The output of LLMs might be congruent with reality and how the prompt contents represent reality. But they might also not be, and in subtle ways too.

If only all code that has any flaw in it would not run. That would be truly amazing. Alas, there are several orders of magnitude more sequences of commands that can be run than that should be run.