Good quick fix for this: put a tiny pad of scotch tape on the flat face of the connector. Don't let it overlap the edges.

The extra friction will almost always hold the connector in the port reliably.