That's definitely not the case with my ISP. I'm routinely using IPsec VPN which uses UDP packets and the only (barely) noticeable difference is +100ms latency. Actually I'm often getting better Internet experience with VPN, presumably because of different routes to VPN server and target website.