I used to feel that way too, but then I started paying closer attention to travel times, which were the same no matter whether that happened. Unless you’re on a rural road with no traffic or signals, a cyclist going 15mph is probably not costing you a second of door to door travel time rather than decreasing the time you spend stopping for signals or waiting for the car ahead of you to start moving.