IP address is involved in some receiver's reputation calculation. It's never involved when sending to a domain.

Sure but then your mail gets dropped on the other end: The main issue I had the last time I tried running my own setup for mails was basically getting an email to an outlook or live.microsoft address. My mails were dropped for no reason, effectively not landing in my friends mailboxes and without any error on my side to know that my mail was getting rejected.

This is when I decided to stop trying getting through with this and came back to paying a provider.