I ran some tests with the guys in WireGuard IRC which seemed to confirm that the issue is specifically EE limiting UDP whether by QoS or otherwise.

I haven't contacted EE about it or tested other VPNs yet. I want to run WireGuard for various reasons so switching to OpenVPN might confirm they issue but not solve my problems (I don't run the VPN for the reasons in the OP)