This will not work. Most of your visitors will depend on some upstream resolver (for example google's resolver), which will cache you for 24h (you can explicitly request a flush - but the whole point of this system is that it's automated). AFAIK (which is a year or two out of date), every big DNS resolver that you might expect someone to use does this, to prevent the enormous performance issues from people setting their TTL to anything less than that.
Edit: Of course if you use a good relay server that isn't owned by you you can mitigate this, because they'll keep the emails in a queue for you until you can be found again.
And this isn't just a record cache. Even a nameserver change seems to get picked up in less than 15 minutes worldwide.
After the TTL expires, the only hits left on the old IP are from China (Great Firewall?).
Email is extremely tolerant of downtime. There are also commercial backup mail servers (these accept and cache the email until your server is back up), which can be used as the only public way to receive email. Some will deliver it to a non-standard port.