They could, but if you have 4 people in a call and 3 can receive high bandwidth audio, lowering it just for the 1 person with low bandwidth is the best user experience.

Otherwise people with good networks who have their call quality dragged down will just think Discords voice chat is bad.