The "Web track" task at the annual US NIST TREC conference ( https://trec.nist.gov/ ) is an open innovation benchmark that everyone can contribute; participants get a set of queries that they have to run on exactly the same corpus. Then they return the top-k results to a team that evaluates them.

Here is an example (2014) Web track paper of the 23th TREC: https://trec.nist.gov/pubs/trec23/papers/overview-web.pdf (TREC has a plentitude of difference benchmark tasks and you can submit your own: https://trec.nist.gov/pubs/trec29/trec2020.html - recent TREC 2020 papers)

(Founder of Neeva) Eval for search engines is about as hard as ranking for search engines. You need rating templates, raters, querysets, tooling and lots of time and patience. There are a number of vendors who can help you with the raters part, but the rest is still painstaking work you have to do yourself. Email me if you need help; we are happy to share findings.

There are benchmarks within the adjacent field of information retrieval, but in general it's hard to properly validate a search engine because real data is so noisy and misbehaved, and sample data is so different from real data.

Sure, the problem of information retrieval is not exactly that of web search but they're pretty close. So, from such a knowledgeable person such as yourself, when it comes to this topic, could you remind us, what are some of those benchmarks?

>> Precision and recall

The phrase I was looking for. Thx a bunch! Gonna marginalia that now.

Haha, ironically it lacks both precision and recall for that topic.