I agree, however with any team with size > 1 you then have to manage access. I've seen small teams share a common credential and getting SSO to work OOTB with that VPN infrastructure in SCM is a pain. Additionally, if you end up having more behind that VPN (e.g., db's, monitoring, etc) its arguably net worse to do than the port-shift IMO.