We used two small Sun E250s to send upwards of a million emails a day using qmail and a tweaked version of the Solaris TCP settings to close connections as quickly as possible. These two servers also handled the bounces as well as the usual spam attempts. If I was doing something similar again I'd still used the Solaris/qmail combo as it was very reliable.
Our biggest problem was with people marking us as spammers even though they had to sign up to groups to use the service but that's a constant headache in this game. We also had some shell scripts that culled SMTP connections (either incoming or outgoing) that had been help open too long as that helped us deal with DOS attacks.
The site also throttled SMTP traffic to give HTTP traffic priority if required as that's OK for SMTP with its store-n-forward architecture but not for HTTP
Solaris is very good, and the x-BSDs often do better if you're a free software fanatic.
Practically you might find your CPU utilization is 2%, but the load average is 50 and you do 'ls' on a directory that has 20 files in in it and it takes 30 seconds.
I don't think there's a big technical issue in scaling SMTP sends; you can have N machines doing it in parallel and it will work (almost) N times faster than one machine. There IS the question of minimizing your cost per unit email, and the real spammers address that by building SMTP senders that don't comply with the standard.
What does bug me about email is deliverability. It's pretty much impossible to send e-mail to AOL members, for instance, if you (1) don't pay ice to AOL, or (2) are big enough that AOL is afraid you'll sue them. AOL has shrunk a lot lately, so this isn't as big a concern as it was years ago.
It's not hard to get burned by other organizations as well. For instance, I sent an email shot on the the behalf of a campus organization at a university from off campus... an opt-in list of 2500 subscribers, really just chicken feed, but only 1999 went through -- after 2000 connection attempts, they firewalled my IP address and for all I know that address is still firewalled today. I knew the guy who runs email for that school by name and refused to talk about the whole affair... That's what you're really up against.