That's the whole beauty with your own domain. They don't have to touch your server at all, it's enough if they can social engineer their way into your account at the DNS provider and point your domain to their own email server. Your security isn't even considered in this case. The only thing that can save you there is how good the DNS security is.