Well, mostly I was thinking about code, and aside from the specific exceptions of trading algorithms (which I was trying to get at when I said hedge fund strategies), and now PhD theses (good point, at least if you're talking pre-publication), I'm still having trouble understanding the threat model even if AI did train on most proprietary, private business code. Can AI training on a CRUD app's code damage a business?

And I have the same question about private notes, or even a diary. Can an AI training on a bunch of personal stuff damage the person that wrote it?

Do you really keep trading algorithms on github?

Well, depends on what you have in those private notes and how others will query the LLMs trained on that private data. Maybe you write things in private notes that are a reason for private notes to remain private.