Ha! Interesting.. I wonder if Google does count that against you since part of their latest algorithm factors in page load times.. though I would think that having your site behind https might give you some extra kudos somewhere.. considering you verified the integrity of your site at least.. (ALTHOUGH, there was that recent security breach where we someone can spoof being a CA...)