That's usually the opposite case, wider apertures mean more light which gives the AF system more to work with. The AF speed of my 135L drops significantly if I put a 1.4x teleconverter on it despite needing to move the same amount of glass.

Most wide-aperture lenses have larger AF motors to compensate with the exception of a few unique lenses that use a gear instead of servo motor(USM in canon parlance).