News for hackers. Stuff that matters.

A comment on Turn entire websites into LLM-ready data. Parent comment

Re:Turn entire websites into LLM-ready data by super256 on Saturday April 13, 2024 @07:42PM (#40027248)
> FireCrawl is built to navigate common web scraping challenges, including reverse proxies, rate limits, and caching

They probably ignore robots.txt

[ Parent | on HN ]

There are no comments on this comment yet.