I had a similar problem with my Nexus 5X. I worked around it as you've described for quite some time before I realized that a small amount of lint had been compressed into the port, which was preventing the connector from going all the way in.