I assume they do hallucinate, just like with coding or finding vulnerabilities.

You can try to minimize it (e.g. with a reviewer agent, which Claude Science and Biomni have), but nothing is perfect. I limit autonomous work to verifiable problems and review it.

Personally, I don't use it to write, and I'm against using current LLMs to write papers.