Analogy time: in the early days of the Internet, any IP address could send an e-mail by directly connecting to the mail change host listed for the domain of the To: address via SMTP. This was a good, nice thing.

Bad agents, spammers, ruined that. The trust is gone, and wow it's a terrible idea to accept SMTP connections from any random IP addresses.

Legitimate senders have to "opt in" to the SMTP sending system by getting a static IP of good repute, or else using a forwarding host which has those attributes.