My argument is predicated on measurable differences of realistic weights of bikes and riders.
That is, would you expect adding ten pounds to have a meaningful difference in how fast a bike will accelerate down a hill? Even fifty pounds?
It will impact brake force needed to stop. But the speeds are going to be near identical.
F = G * m1 * m2 / r²
This tells you that the acceleration of mass 1 is independent of the mass 1 and only depends on mass 2 and the acceleration of mass 2 is independent of mass 2 and only depends on mass 1. But the relative velocities will increase when either m1 or m2 is increased.
E.g. a black hole with the mass of the earth will hit the ground of a (perfectly rigid) earth in almost half the time as a feather would.
(I won't go into relativistic effects here.)