The first barrier was that homoglyphs would inhibit text search, so I had to build an automated homogylph detection and substitution layer.
Once homoglyphs were stripped, the challenge was then to fit the entire search corpus into memory, so I compressed each page with LZMA, loaded it into memory, and decompressed on the fly when searching—probably not optimal, but still way faster than loading from disk.
I always wanted to try reverse engineering some of the watermarking systems so we could modify the watermarks on certain material, subtly leak it, and effectively frame adversaries while protecting our own spies in the process. Fortunately or unfortunately I never got around to that.
For forum posts, an adversary can do exactly what you say - just take the meaning of the forum post and write a cable on it.
Fleet pings are how massive fleets are formed in EVE. Basically they are a @here ping on Discord, or a server announcement on XMPP or IRC. Fleet pings are important because if you know when your enemy is pinging for fleets, you know when and where to find their fleets to fight them.
For that reason, the value in fleet pings is very time sensitive, and cannot be transcribed by a spy in real time. Usually orgs will set up a "ping relay" channel for its top fleet commanders, with a bot that relays pings from enemy alliances. This is a very common interception point for obtaining watermarked pings and burning spies.
Such an interesting game. I played for years and always want to go back.
One group decided that writing an automatic transposition program was too hard, and instead manually create transpositions for major posts. You can find the writeup here: http://failheap-challenge.com/showthread.php?16311-Taking-th...
Would love to see a swing taken at Pandemic Legion's new forum background watermark, it's a lot more interesting than the old one.
Some were identified and others... not
Zero-Width characters tend to cause lots of issues when they are copied and pasted, which may alert a poorly equipped adversary that they're handling watermarked content. In addition, entities that are aware that you're watermarking text content in this way can just take screenshots of the text, transcribe it, or strip it of all non-ASCII characters.
The best solution I've seen is something I like to call "content transposition". The idea is that you take a paragraph of content and run it through a program that will reorder parts of content and inject/manipulate indefinite articles in order to create your watermark, while keeping the content grammatically correct. That way even if an adversary is fully aware that you're watermarking text content, they need two copies of the watermarked text in order to identify and strip your watermark.