I believe that sites that are served securely are ranked higher in Google search results, so you're actually ranked lower now than if you'd already gone through the process of securing your site.

Also, millions of sites have already secured themselves and survived just fine. I just don't know what you're worried about. Can you point to some links of this having been a problem for other websites?