Vanilla qmail was never an open relay unless you really intended to configure it as one. It's just that the only relay controls were IP-based. Back in the day, before AUTH was widely supported, I used a really nice "relay-ctrl" add-on from Bruce Guenter for POP-before-SMTP.
For mandatory user-facing TLS and AUTH on port 587, and opportunistic server-to-server TLS on port 25, I've written https://schmonz.com/qmail/acceptutils. For SMTP recipient checks, I've written https://schmonz.com/qmail/rejectutils. It's too soon to say how notqmail will solve these problems, but they're solvable and we intend to solve them. For DANE... definitely not there yet.
So, yeah, qmail is fine as a starting base, but there are important things that a modern MTA/MSA needs to have:
There's more stuff too (delivery options, filtering, ...), but protocol-wise, the above are critical.I'm not sure that portability is such a problem that you can't use Rust. Rust runs on all modern OSes, which is all you need to support in a new MTA/MSA. And Rust is fairly easy to get into, actually.