The speed-up has another component: semantic compression: you express more in a single embedding vector vs. tokens retrieved.

Your point is valid for the agentic tool search, i am just saying there are two things lurking.