Also did you mean every single wikipedia site or just the English one? I'm thinking just the English one's sufficient.
I still don't understand why wikipedia articles aren't translated according to a common script, at least where there are featured translations (marked a star)
As for anyone curious:
https://dumps.wikimedia.org/enwiki/20180320/
The article text contains Wikipedia markup which is a bit difficult to remove but not impossible, there are some existing projects for doing that. DBPedia would have the raw text, but it's not nearly as current.
http://dumps.wikimedia.your.org/other/cirrussearch/
These are structured with a JSON string for each doc roughly like https://en.m.wikipedia.org/wiki/Foobar?action=cirrusdump
Our complaint is that the results we get from our queries simply do not match the queries we are making.
I'm imagining that "the frontier" a crawler needs to crawl is actually a distributed queue and that crawlers are massively parallelized. I'm also imagining the frontier is bucketed by its indexing frequency i.e daily, weekly monthly etc. Is that close? Might you have any resources on how this is architected at large search providers?
[1] https://venturebeat.com/2013/03/01/how-google-searches-30-tr...
Certain pages get crawled more often than others. As a website owner, you can tell Google how often your content changes, which they will use as a clue for how often to recrawl your site.
If you're really big (like for example Reddit), you actually lose that control in Google dashboard -- they fully control the crawl rate. From experience, I can tell you that they are crawling large sites, like Reddit, continuously. Even 7 years ago, when I last worked there, they were crawling Reddit so much that we had to set up a separate server infrastructure just to respond to Google's requests, because their access patterns were so different from every other user.
> "[H]ow is the crawling ever completed before
> it needs to be re-indexed again?"
The reality is that there are roughly between 5 to 15 billion pages that are nominally "not spam" and not duplicates. Literally 99.9% of the internet is crap. So finding web pages as long since switched from 'crawling every page that is accessible from the web' to 'only surfacing those pages that have something of value on them.' That was the fundamental thesis for the founding of Blekko and it is still true today.
That said, a cluster of ~100,000 threads and a couple of petabytes of storage attached with sufficient bandwidth to keep the threads busy can deal with what is out there. If you can create a hash space for strings that is sufficiently uniform you evenly spread the load of crawling every URI you discover across the cluster.
As you crawl you take new pages you discover and apply your ranking algorithm to them where they will score a value between 0 (never index) and 1 (always index).
At which point you can dial the 'rankable' value from 0 (index everything) to 1 (rank only must rank pages) to set the size of the index you can tolerate.
I think what OP and several people in this thread actually want is Google search minus synonyms and the ability to specify advanced syntax like AND and NEAR queries. I believe that would go a long way to satisfying someone who says they just want "keyword search".
These days its fairly easy to code up an n-gram index and host it on a moderate server. If you have a corpus of documents you can see how well it works. The simplest corpus is to just down load all of Wikipedia. Last time I checked it fit on a 2 TB disk drive.
You could also use the Common Crawl database and index it as you would like, or talk to the Internet Archive project for some sort of collaborative project. However I will warn you that of the people who have done this experiment (and it was a new hire task at both Blekko and Google where I worked) folks quickly discovered it wasn't very useful.