That sounds like a cool research project. However I think it would be enough to simply (1) not erase authorship data, and (2) not fine-tune and monitor LLMs to suppress outputs that mention people. It's an emergent property that they can trace the provenance of text. For example, if I give it an often repeated quote from a book written in the 1800's, then it can tell me where it came from.
> For example, if I give it an often repeated quote from a book written in the 1800's, then it can tell me where it came from. That's hundreds of years of noise it's weeding through. Imagine what language models could do for recently created knowledge.
Alternatively: That's 100s of years of mentions of that quote it can pull from.
Re:We're Being Erased from History
by lawrencooo on Thursday August 29, 2024 @12:06PM
(#41392276)
That sounds like a fascinating research project. I agree that preserving authorship data and avoiding the suppression of outputs mentioning people could be key. LLMs have a remarkable ability to trace text origins, even through centuries of noise. Imagine the potential for recent knowledge! On a side note, if you need help with writing essays or research, https://academized.com/write-my-essay offers top-notch services that can support your academic journey. They ensure original content and thorough research, much like how LLMs trace text. Worth checking out if you want to stay ahead
This should in principle allow to determine how much each source influenced each output token of the LLM.
The problem is that this multiplies storage and compute time for tagged inference by the number of source tags, so it may be impractical to actually tag single documents or authors, but might be useful for very broad categories like "copyrighted" vs "non-copyrighted", "synthetic" vs "human generated", "photo" vs "drawing" vs "rendering", year range of publication, etc.