I find hallucinations in my (mostly perfect) AI output every single day. If you're not finding them, you're just not looking hard enough. It's not surprising when everyone is screaming about how they don't read code these days.

This is just a fact. I'm sorry if it messes with your narrative.

https://arxiv.org/abs/2401.11817

Thanks for the link. What kind of hallucination are you seeing, and does it affect the end result?

I'm not them, but I see quite a few hallucinations as well. Some examples:

I work on a device that gets firmware updates over USB-DFU. If a DFU fails, its bootloader restarts, re-enumerates USB, and waits for a new DFU to begin. AI decided there's a risk of bricking the device if an update fails. There's no brick risk, a retry fixes it.

The same device uses CAN bus, with the common bosch_mcan peripheral. Claude decided that calling can_mcan_stop followed by can_mcan_start somehow left the transmit buffers intact, and tried to implement a fix which manually cleared the buffers. They're cleared automatically by the hardware when can_mcan_stop is called, and that's documented in the comments of can_mcan_stop.

Both cases could have resulted in unnecessary changes getting deployed which wouldn't have fixed the actual issues. Since it's embedded that could mean significant delays to getting fixes to customers.

The sum total of all human observations is still not proof of the lack of hallucinations as a problem (even if their observations were perfect, which they aren't considering the volume produced vs reviewed carefully). That's why you can use a counter example only to disprove and not prove anything.

And yeah I get hallucinations all the time still. Maybe it's because I'm working on harder/more niche problems (like a compiler with an unusual type system), but it happens quite a lot. I don't record all of them.

Although the most common one you can find is them misattributing the source of changes from themselves and also other agents (Fable, Opus 5.5, deepseek, whatever). They'll say "your changes" or "you changed" or "your ruling." I didn't decide anything and it's in their own chat log, and yet...