News for hackers. Stuff that matters.

A comment on Turn entire websites into LLM-ready data.

Re:Turn entire websites into LLM-ready data by altdataseller on Saturday April 13, 2024 @07:39PM (#40027231)
What user agent does it use and do you obey robots.txt?

* apparently not as I tested it with LinkedIn.com which blocks most non-google/bing bots and it crawls it OK

[ Parent | on HN ]
Re:Turn entire websites into LLM-ready data by super256 on Saturday April 13, 2024 @07:42PM (#40027248)
> FireCrawl is built to navigate common web scraping challenges, including reverse proxies, rate limits, and caching

They probably ignore robots.txt

[ Parent | on HN ]