It's infuriating when Google prefers the documents that ignore some of my keywords even when there are plainly pages that include them all. I get the sense that there's an over-weighting of broad semantic match, to the detriment of lexical match, in their current ML model, whatever form it is now. It's harming quality for a segment of technical users like ourselves but might be "better" in the aggregate over all users, according to at least some of their internal quality metrics.

Back in the 90s, search engines were driven largely by sparse vector representations of the documents such as TF IDF vectors before latent semantic indexing, topic modeling, and other dense vector representations like sentence embeddings entered the fray (not to mention non-content features that use the web graph, click stream, etc.). A lot of NLP applications use a mix of dense and sparse features but it's hard to get the balance right in a way that works for all inputs. Google's pendulum has swung too far in the "dense" direction, as it certainly seems "dense" a lot more often lately!