presumably, even in hands-free mode the attacker needs to be very close to the speaker to hear it

If you have a Bluetooth analyzer (e.g. Ellisys), then the link key and a directional antenna is all you need to passively eavesdrop on a conversation at a distance.

Of course, even regular omnidirectional Bluetooth antennas are plenty to eavesdrop through a hotel room door, from the hallway outside a conference room, etc.

An attacker can also passively record all the packets in an area (Ellisys allows recording all channels at the same time), and then actively gather link keys using this attack at any time to decrypt the stored conversations.