The core function of that is actually pretty simple:
1. Strip all X/HTML tags 2. Run `diff`
Sure, it's not perfect, but an organization that pursues academic quantum computing research can sure as hell afford to run the results of the above against an AI to check for similarities.
In the case of things like other sites copying from stack overflow, github, etc. you can figure out which is the source of the information.
Lets day you have decided to make github the source of that information, and derank any other sites that also have that information. As a result, you will derank gitlab for having git mirrors of projects, as the source will match that on github. You will also derank sites like lkml as that contains the commit message descriptions, and the patches will partially match content from the kernel source hosted on github.
Lets also say you've decided to make wikipedia and other wikimedia sites the source of information. Congratulations, you've now deranked project gutenberg for hosting the same public domain texts as wikimedia, along with other sites like the official sites for authors like Jack London. Plus any blog that includes portions of these to discuss or analyze them.
Do you want a blackbox AI making these decisions?
I interviewed with a 3 or so person company that did just that I think.
That was over a decade ago.
So they could know if a code snippet was already X years old? And from where, originally. (Unless it got edited a lot but that's not the case here?)
Related is if you have a lot of content from git repositories that are mirrored from different locations (GitHub, GitLab, etc.), all of which are showing the same content. Or if different sites are hosting versions of public domain texts. You don't want to derank those results, even if they are similar to the "copy a popular site" websites.