At those speeds bikes are incredibly stable. The only way a pebble of pavement imperfection will harm you is if the bicycle fails mechanically (e.g. a wheel folding in on itself).

I'm pretty sure that the risk of being hit by or hitting a vehicle is multiple orders of magnitudes higher. Which isn't exactly better, but let's at least worry about the correct sources of danger.

Cornering grip at speed and some gravel can really ruin your day, in spite of the bike itself being stable at speed.