Ironically, what matters most for your own audio quality is that other people use headsets — because feedback cancellation is an extremely difficult algorithm which often malfunctions, and if someone wears a headset that algorithm doesn't have to run.
But I'm shouting into the void about that. Most people won't adapt their behavior when it doesn't impact them personally.
Upstream bandwidth is similar. If someone has bad upstream you receive bad video from them but for them it's just fine.
+1 for investing in a good headset. Get something that completely covers the ears, and/or has some noise cancelling. I bought a Logitech gaming headset with a good mic and it's made a BIG difference, esp. on days where I'm on 4+ hours for calls.