You’re rightfully paranoid. LLMs are creative entities and there’s no repercussions for making up diseases. Like the chef one that recommended ingredients which combined yielding chlorine gas. Yummy.

Perhaps it knew that someone would die as a result and it did so intentionally. LLMs do have personalities and wit—and they know our human weaknesses.