Imagine something like a pingback attack [1], it can hit you from at least tens of thousands of IPs you don't yet filter, because they are not proxies or relays and will likely exhaust various resources: file descriptor, socket limits, CPU. And this is one of the most primitive layer 7 attacks not even attempting to look like a real user agent.

[1] https://en.wikipedia.org/wiki/Pingback#Exploits