Re:A Look Inside Our 210TB 2012 Web Corpus
by Sven7 on Wednesday August 14, 2013 @07:26AM
(#6210887)
Which is one of the main reasons Google has no serious competition in search except possibly in China.
It is also why the rate of innovation in search isn't moving as fast as it can be moving.
If Google opened up (unlimited) web API access to their search interface, to say a large city for a year or two people would really get a taste of what innovation in search looked like.
And of course it would be in Google's interest cause search as a platform or marketplace is where the future of Google really lies. All the other advertising empire defending distractions like Android, Chrome and YouTube are really sideshows.
Re:A Look Inside Our 210TB 2012 Web Corpus
by boyter on Wednesday August 14, 2013 @07:31AM
(#6210905)
Personally I consider extracting meaning from the crawl part of the indexing step. That just comes down to how you define it though. In reality its blurry as you need to do some pre-indexing during the crawl to extract meaningful data and as you say there are a lot of edge cases.
For basic crawling it really is as simple as while links download link though.
What I heard about a smaller search engine was that web crawling is usually augmented with some manually added rules for various sites to prevent spoiling the database. Not a trivial task at all.
Doing queries is IMHO algorithmically much better understood, because it's a constrained problem. But getting information extracted out from the real world, with all the PHP and HTML "hackers", not so easy.