Trying to apply those rules to drone deliveries while ignoring those two pretty fundamental elements doesn’t strike me as evidence that the FAA is being arbitrary or capricious.
When you're risking your own life, you'll likely be at least somewhat careful.
When you're just flying an empty test vehicle you'll be much more reckless, and thus way more likely to crash it on other people's property or other people.
Hence, much more tightly regulated.
But they have limits:
No passengers allowed.
No flying over towns or settlements.
No flying at night or above (or in) the clouds.
No flying in airspace around airports with control towers and certain other airspace without prior permission.
No commercial operations (for hire) except instruction.
Ultralights must yield right-of-way to ALL other aircraft.