Upgrading the distro is likely insufficient, as your private key has potentially (likely, at this point) been compromised. At a minimum you need a new key and cert.