Yeah - as someone that has run production search clusters before on technologies like Elastic / open search, deep pagination is rarely used and an extremely annoying edge case that takes your cluster memory to zero.

I found it best to optimize for whatever is a reasonable but useful for users while also preventing any really seriously resource intensive but low value queries (mostly bots / folks trying to mess with your site) to some number that will work with your server main node memory limits.

I guess they have multiple search interfaces.

I navigated to loc.gov and searched directly from the home page.

I was sent to this page: https://www.loc.gov/search/?in=&q=god&new=true&st=

At the top, it says “Results 1-25 of 324,782” and does not mention the limit of 10,000 anywhere.

It does once you get to the 4,001st page of the results (which is actually 100k items): "Sorry! We can't process this request. This request exceeds the maximum search results depth." The error page even includes a link to "LC for Robots" (https://labs.loc.gov/lc-for-robots/), a list of APIs, since few humans are going get that far on their own.

The number here also seems less misleading because all 324,782 results do seem to exist. It doesn't want to generate a pagination for the entire set, but you could get to them by choosing different formats, date ranges, collections etc. The number Google repors, as far as I can tell, needs to be taken on faith.

Ahh, looks like the limit on results is 100,000 items on that page. https://www.loc.gov/search/?q=god&sp=4000

But you can subdivide the search results by date and get every item on your original list: https://www.loc.gov/search/?dates=1890/1899&q=god&sp=1808

So, they have and can show you every single one of those 324,782.