I used to do something similar: I hosted my own IMAP, while using Gandi for receiving mail and sending mail. That meant I didn't lose mail on the receiving side if my mail server had an issue, and I didn't fail to deliver mail to others because I hadn't jumped through enough hoops. (Literally everyone I personally know who has run their own mail server has had one or both of those problems at least once.)

This worked well for me because it gave me the feeling of having more control and privacy and security over my email.

I switched away from that solution when I realized that in practice I have less ability to effectively provide security than the whole security and product teams of a major email provider.