That's my experience from running with regular earbuds. Have to find a way to hold my phone just so to keep the cord from jostling them out. I've also tried a phone arm-strap but couldn't work out a place that worked for the headphones cable.

Don't have airpods, but I've heard nothing but great reviews so I'm thinking about it. Wish they didn't cost so much.