I have the same problem, clearly I have weird ears. They're a little cumbersome but I bought some EarSkinz and the problem is 100% solved. https://www.amazon.co.uk/gp/product/B0741CXH71/ref=oh_aui_se...
The audio quality is also massively improved because of the better fit. The downside is you have to take the skins off before putting the AirPods back in the case. I also bought one of these pouches to keep the AirPod case and EarSkinz in. Works well, all in all
Love them in general though, they are good enough for the price & what they do.