Yeah, integrating identity providers could work, knowing recipient email addresses and that they use particular services. However that has some complexity/libraries and as with that - vulnerabilities all over again. I just don't want to think/expose attack surface and give possibilities for automated attacks to leak personal data.
Or lets say it like that - I trust iptables more than myself integrating identity provider or depending on some library.
Actually this is what onedrive/photo galleries etc. does when you share files/folders - generates some magic link that penetrates auth. However it's the same story - I trust complex system (that may have well known vulnerabilities and I was on a vacation) less than iptables.
I would rather the server doing any processing of the packets be running as a low privilege user than sitting in the kernel.
If you want that, just run the webserver and base the response on IP address (have your 401 forbidden page say "forbidden", and allow from 1.2.3.4). Or do a username/password, or do something like openauth to integrate with an identity provider