I really doubt that that factor has any real influence on ranking. In my niche there are notoriously slow websites at the top that aren't even responsive in some cases.

Just like the promised benefits of HTTPS. I have yet to hear about a case where search traffic increased after switching to HTTPS.