It's a trope from the days when you could still get vehicles with manual brakes with no power assist. As you can imagine enabling such a vehicle to go faster greatly increases the workout the operator is going to be getting to the point where if you want to drive aggressively in stop and go traffic the operator is the limiting factor.