sammitch
Re:A Facebook crawler was making 7M requests per day to my stupid website
on Thursday June 11, 2020 @07:18PM (#23494238)
Plenty of crawlers, including Facebook's, are operating out of huge pools of IPs usually smeared across multiple blocks. If you have an idea on how to ratelimit a crawler doing 2 req/s from 300+ distinct IPs with random spoofed UAs let me know when your startup launches.