I'd wager it's because the most relevant results for a search don't become less relevant just because they load more slowly. I certainly don't want that as a user.

Website performance could be a minor tie-breaker between two similarly ranked websites that have the same information (and it probably is), but I don't see how it could be anything more than a weak signal in a search system where the objective is to find the most relevant results.

I'll wait 30 seconds if it means I'm loading the result I'm looking for. That a less relevant website loads in one millisecond is frankly immaterial to me.