In my experience vector search (top 50 results) combined with reranking (top 5-15 of those 50 results) yields not only great results but is even quite performant if done right (which is not hard!).