Re:“The world's most private search engine”
by ChuckMcM on Friday June 21, 2013 @11:08AM
(#5918387)
In that particular case it gets a lot easier of course. One of the challenges of "general" search is that you're trying to satisfy a wide variety of queries, for domain specific search you can be much more selective of what you pick. Also one of the 'knobs' if you will is change in the corpus, so if you index Wikipedia for example it doesn't change all that much and that is a pretty stable index, versus if you index say a few tens of thousands of blogs then they will get new articles to index once every .5 to maybe 3 days. Versus keeping an eye on the "news" sites which get perhaps a dozen articles a day added to them.
Then if you don't keep the original extractions around, just holding the index can be done on a relatively small number of machines (say 5 or 10) and for a small user population that is pretty doable.
Google sold its "Google Appliance" which did this for businesses and often that was less than 1 cabinet or rack worth of machines.
Given some of the silly things people try to scrape Blekko for (clearly domain specific things) I'm surprised there aren't more folks building such engines. It isn't hard to write a basic crawler, it takes a lot of coaching to get it through the initial crawl since if you just blindly follow links around you can end up in endless loops on some sites or in a quasi form submission loop (we call those Crawler traps, for the most part unintentional on the part of web sites [1])
And of course we have been supporting the Common Crawl efforts to make this more generally available. We gave them a bunch of indexing data that made the Crawl they had a lot more useful as an example. So at some point we may reach critical mass and get that going.
[1] Sometimes though folks use an intentional crawler trap to find 'stealth' scrapers, which are folks making their own crawlers and not respecting robots.txt.
Then if you don't keep the original extractions around, just holding the index can be done on a relatively small number of machines (say 5 or 10) and for a small user population that is pretty doable.
Google sold its "Google Appliance" which did this for businesses and often that was less than 1 cabinet or rack worth of machines.
Given some of the silly things people try to scrape Blekko for (clearly domain specific things) I'm surprised there aren't more folks building such engines. It isn't hard to write a basic crawler, it takes a lot of coaching to get it through the initial crawl since if you just blindly follow links around you can end up in endless loops on some sites or in a quasi form submission loop (we call those Crawler traps, for the most part unintentional on the part of web sites [1])
And of course we have been supporting the Common Crawl efforts to make this more generally available. We gave them a bunch of indexing data that made the Crawl they had a lot more useful as an example. So at some point we may reach critical mass and get that going.
[1] Sometimes though folks use an intentional crawler trap to find 'stealth' scrapers, which are folks making their own crawlers and not respecting robots.txt.