Turn entire websites into LLM-ready data

Posted by nickca on Saturday April 13, 2024 @06:32PM
from the firecrawl.dev dept.

( 11 comments | 13 points | on HN )

Re:Turn entire websites into LLM-ready data by altdataseller on Saturday April 13, 2024 @07:39PM (#40027231)

What user agent does it use and do you obey robots.txt?

* apparently not as I tested it with LinkedIn.com which blocks most non-google/bing bots and it crawls it OK

[ Parent | on HN ]

Re:Turn entire websites into LLM-ready data by super256 on Saturday April 13, 2024 @07:42PM (#40027248)

> FireCrawl is built to navigate common web scraping challenges, including reverse proxies, rate limits, and caching

They probably ignore robots.txt

[ Parent | on HN ]

Re:Turn entire websites into LLM-ready data by padolsey on Saturday April 13, 2024 @07:54PM (#40027341)

Do you have pay-as-you-go pricing? I feel that’s always missing from things like this. Cool otherwise.

[ Parent | on HN ]

Re:Turn entire websites into LLM-ready data by jshchnz on Saturday April 13, 2024 @07:37PM (#40027222)

This is cool, seems like a nice way to easily add context for ChatGPT at the least

[ Parent | on HN ]

Re:Turn entire websites into LLM-ready data by cpeffer on Saturday April 13, 2024 @07:44PM (#40027261)

* Creator here - Thats the goal!

[ Parent | on HN ]

Re:Turn entire websites into LLM-ready data by donohoe on Saturday April 13, 2024 @08:16PM (#40027497)

And you honor or ignore robots.txt?

[ Parent | on HN ]

Re:Turn entire websites into LLM-ready data by cpeffer on Saturday April 13, 2024 @08:27PM (#40027557)

It wasn't in our initial version (we didn't plan on launching today), but we are pushing an update to do so now.

[ Parent | on HN ]

Re:Turn entire websites into LLM-ready data by pryelluw on Saturday April 13, 2024 @07:51PM (#40027315)

Does this respect any anti AI related scraping rules set forth by website owners?

[ Parent | on HN ]

Re:Turn entire websites into LLM-ready data by duskwuff on Saturday April 13, 2024 @08:02PM (#40027403)

It doesn't even respect general robots.txt exclusions, so... I doubt it.

[ Parent | on HN ]

Re:Turn entire websites into LLM-ready data by nyolfen on Saturday April 13, 2024 @07:51PM (#40027317)

[ Parent | on HN ]

Re:Turn entire websites into LLM-ready data by cpeffer on Saturday April 13, 2024 @07:55PM (#40027352)

It crawls webpages (finds subdirectories), handles JS blocking with fallbacks to headless browsers, and does this all concurrently.

If only that script worked for every website. But, alas, it does not.

[ Parent | on HN ]