That's a factor, but I think it's more that there was never a need for large data uplinks to Voyager, while they needed to downlink large amounts of science data.

The uplink and downlink are in different frequency bands, S-band (2-4ghz) for uplink and X-band (8-12ghz) for downlink and have different amounts of radio frequency bandwidth allocated.

Additionally, Voyager's High-gain antenna has a much narrower beamwidth in X-band at 0.5°, compared to 2.3° for S-band, though I have no idea if that's a factor.

Plus receiver size.

It’s mainly this and it’s normal for spacecraft (especially in deep space) to have orders of magnitude more datarate on downlink.