News for hackers. Stuff that matters.

A comment on Launch HN: Exa (YC S21) – The web as a database.

Re:Launch HN: Exa (YC S21) – The web as a database by jackienotchan on Tuesday May 06, 2025 @03:15PM (#43908630)
AI crawlers have lead to a big surge in scraping/crawling activity on the web, and many don't use proper user agents and don't stick to any scraping best practices that the industry has developed over the past two decades (robots.txt, rate limits). This comes with negative side effects for website owners (costs, downtime, etc.), as repeatedly reported on HN (and experienced myself).

Do you have any built-in features that address these issues?

[ Parent | on HN ]
Re:Launch HN: Exa (YC S21) – The web as a database by antoniojtorres on Tuesday May 06, 2025 @03:45PM (#43908892)
I work in the adtech ad verification space and this is very true. the surge in content scraping has made things very very hard in some instances. I can’t really fault the website owners either.
[ Parent | on HN ]