Re:Google’s “million’s of search results” are not being served
by acdha on Monday October 03, 2022 @11:02PM
(#33076298)
> EDIT: I just searched the Library of Congress for “god” and was told I was viewing results 1-25 of 324,782. Sure enough when I asked for page 10,000 I was rebuffed. I really don’t think this is a Google thing
This is a common problem called deep pagination, and it basically stems down to how most search systems work. When you make that search request on loc.gov, it eventually turns into a complex request for Apache Solr. Answering that request requires Solr to find all of the matching documents which match the query and any other active filters (e.g. document type, date, etc.), and calculate their rank in the search result set so it can determine which specific results are numbers 10,000-10,050. Solr/Lucene are really mature and well-tuned but when you think about how many millions of documents there are in that index, this is still calculating a pretty large temporary set only to serve 50 records, throw away the other 324k, and go on to the next query. Your next request might arrive a second later but get routed to a different replica so it’s a cold cache there, too. Click one facet on the filter list, and the search engine has to recalculate the matching documents again.
Don’t forget that you’re often seeing the results of more than just the metadata. You often get the best search results by treating entire books, newspapers, etc. as a single group for the purposes of ranking and then displaying it as a single entry on the top-level search. I wrote about this a while back: https://blogs.loc.gov/thesignal/2014/08/making-scanned-conte...
There’s an alternative which is considerably more efficient: cursor-based pagination, where instead of saying count=50 offset=10000 you say after=<unique ID/time stamp on the last result>. That allows the search engine to efficiently ignore all of the records which don’t match that, avoiding the need to calculate their rank, but comes at the cost of breaking the pagination UI: sites like loc.gov give you numbered lists of pages which humans like but you don’t have a way to calculate those after= values so your faster search isn’t as popular with advanced users.
That last part gets around to the other big factor in decisions: relatively few people do this. Most of the time, people are going to adjust their search if they don’t find what they’re looking for in the first few pages. What does do this a lot are robots which crawl every URL on a page, generating millions of permutations for every filtering option you offer. Some of those robots honor robots.txt (perhaps even correctly implemented), some have accurate user-agents, etc. but many do not and basically everyone offering a search service with a large corpus ends up having to make decisions about rate-limiting, limiting pagination depth, capping the number of filter options, etc.
This is a common problem called deep pagination, and it basically stems down to how most search systems work. When you make that search request on loc.gov, it eventually turns into a complex request for Apache Solr. Answering that request requires Solr to find all of the matching documents which match the query and any other active filters (e.g. document type, date, etc.), and calculate their rank in the search result set so it can determine which specific results are numbers 10,000-10,050. Solr/Lucene are really mature and well-tuned but when you think about how many millions of documents there are in that index, this is still calculating a pretty large temporary set only to serve 50 records, throw away the other 324k, and go on to the next query. Your next request might arrive a second later but get routed to a different replica so it’s a cold cache there, too. Click one facet on the filter list, and the search engine has to recalculate the matching documents again.
Don’t forget that you’re often seeing the results of more than just the metadata. You often get the best search results by treating entire books, newspapers, etc. as a single group for the purposes of ranking and then displaying it as a single entry on the top-level search. I wrote about this a while back: https://blogs.loc.gov/thesignal/2014/08/making-scanned-conte...
There’s an alternative which is considerably more efficient: cursor-based pagination, where instead of saying count=50 offset=10000 you say after=<unique ID/time stamp on the last result>. That allows the search engine to efficiently ignore all of the records which don’t match that, avoiding the need to calculate their rank, but comes at the cost of breaking the pagination UI: sites like loc.gov give you numbered lists of pages which humans like but you don’t have a way to calculate those after= values so your faster search isn’t as popular with advanced users.
That last part gets around to the other big factor in decisions: relatively few people do this. Most of the time, people are going to adjust their search if they don’t find what they’re looking for in the first few pages. What does do this a lot are robots which crawl every URL on a page, generating millions of permutations for every filtering option you offer. Some of those robots honor robots.txt (perhaps even correctly implemented), some have accurate user-agents, etc. but many do not and basically everyone offering a search service with a large corpus ends up having to make decisions about rate-limiting, limiting pagination depth, capping the number of filter options, etc.