(Founder of Neeva) Eval for search engines is about as hard as ranking for search engines. You need rating templates, raters, querysets, tooling and lots of time and patience. There are a number of vendors who can help you with the raters part, but the rest is still painstaking work you have to do yourself. Email me if you need help; we are happy to share findings.