I know exactly why I "didn't progress" and they did, as you kind of put it.. I decided not to. They decided to continue. I stopped training, they didn't. When I was still active I progressed same as they did - and I saw nothing super-human in their later performance. Obviously I only have intimate knowledge about my own field, I know nothing about e.g. 100m sprinters, jumpers and the like. And, as everybody else, I'm pretty suspicious about pure muscle-based sports like weight lifting.
That's a ridiculous claim because it can't be falsified as long as you are not arguing with an unbeaten olympic champion. As long as an athlete has been beaten by somebody, you can always pull the "yeah well you know why you didn't win? Because you're clean, but everybody above you doped."
Just like arguing "well we only lost the election cause the other side cheated."
It's simply physiologically impossible to accomplish what these athletes are doing without assistance.