You may lose access to your Google account. I run my own mail server and honestly despite always hearing that people have problems when doing this I've had 0 trouble in years.