Re:Show HN: Generate a free visual sitemap by crawling any w...
by js4ever on Tuesday June 15, 2021 @02:46AM
(#27512104)
you should rate limit this, or it's a DDOS as a service ...
[ Parent
| on HN ]
Re:Show HN: Generate a free visual sitemap by crawling any w...
by owor on Tuesday June 15, 2021 @02:54AM
(#27512151)
We use an auto-throttling algorithm for the crawler to ensure we are a 'polite' crawler!
We also send a unique user-agent with every request, so any host can easily block us if required - luckily, this has never happened so far.