You're mixing up two different things. Transport latency of WiFi is 2-3ms, but for bluetooth you're suddently talking about a transport, codec, receive buffer and playback code latency. That's not an apple-to-apple comparison.
If you want comparable numbers, measure the latency of audio being streamed from one computer to another via PulseAudio and you'll quickly see that it's more than 2-3ms. And there's a reason for that: most people strongly prefer their music to not stutter and work properly with all kinds of headphones instead of having absolutely no latency. Most people prefer to shove their phone in a separate room, back pocket and walk around RF saturated areas and still listen to music without interruption.
Hence the manufacturers err on the side of buffering some audio so it's more resilient to connection dropouts. You wifi, after all, doesn't need to move around busy highways and public transport systems.
I don't immediately see the relationship however -- is there anything you could point me to, to explain the mechanism?
Is it intentional to ensure there's plenty of buffer so audio packets can be re-sent when they fail, and the audio is seamless? In other words, is 100-200ms just inherent at that frequency with expected interference?
Or is it actually just terribly designed?