Agreed (naturally), but also be aware that the relationship may not be linear. There is quite possibly a "knee" in the curve where faster than some threshold will all be the same.
However, it's a quite easy test to run, so may be worth doing even if you can't stake your life on the results applying to a hypothetical speedup.
Before working hard to optimize speed, if you have a working site, try slowing it down and seeing if that makes a difference!