Re:Google search takes 7 seconds on certain queries
by fxtentacle on Sunday August 04, 2019 @05:36AM
(#20605413)
To me, it looks like they introduced bloom filters to pre-filter their indexing. Basically, they run an offline job to create a list of the top 1 mio search keywords and hash them into a compact bitfield. That one is sent to all the crawlers, who will then only index the keywords that have hashes matching the bitfield. It's a standard way to reduce the amount of data that you need to process, and since search keyword revenue is likely exponentially distributed, they would lose almost no revenue by cutting off the 1mio +1th keyword, but greatly reduce their costs.
As for the sentence search, it appears to me that they are using stemmed shingles. For that, you reduce words to their base form (houses -> house) and then you chop the page down into every 3-pair of words and hash those. The hashed number for 3 words is then called one shingle. Again, that's a common technique for content similarity detection, so Google kind of has to do that anyway to spot verbatim copycats. One can then also use this for sentence search by searching for all the shingles in the search query, but that introduces ambiguity. For example "cats singing in front of the house" would likely have the same shingles as "cat sang about front before houses", as only "cat sing front house" are being indexed. The issue here is that "front" can be used both in a positional way, as well as for a military formation.
To me, both of these look like side-effects of them reducing operational costs. I mean, who would bid marketing dollars on "ISpatialAudioClient::GetMaxDynamicObjectCount" ? Given that you will likely tolerate 2-3 complete search failures before trying Bing, the business value of completing such a query correctly is very close to 0.
As for the sentence search, it appears to me that they are using stemmed shingles. For that, you reduce words to their base form (houses -> house) and then you chop the page down into every 3-pair of words and hash those. The hashed number for 3 words is then called one shingle. Again, that's a common technique for content similarity detection, so Google kind of has to do that anyway to spot verbatim copycats. One can then also use this for sentence search by searching for all the shingles in the search query, but that introduces ambiguity. For example "cats singing in front of the house" would likely have the same shingles as "cat sang about front before houses", as only "cat sing front house" are being indexed. The issue here is that "front" can be used both in a positional way, as well as for a military formation.
To me, both of these look like side-effects of them reducing operational costs. I mean, who would bid marketing dollars on "ISpatialAudioClient::GetMaxDynamicObjectCount" ? Given that you will likely tolerate 2-3 complete search failures before trying Bing, the business value of completing such a query correctly is very close to 0.
Microsoft, on the other hand, has a stronger interest in people finding MS APIs, which is why you will find this result on Bing https://audiophilestyle.com/forums/topic/55776-software-to-s... but not on Google.