This looks interesting, how would you compare it with colly? I’ve used colly for a while and overall am pleased with it, but things can always be improved.

Colly is a great library for builing scrapers if you are familiar with Go.

Flyscrape a cli tool that lets you define your scraping logic with a declarative JavaScript API, while it takes care of issuing requests, caching, rate limiting and more.

Ah lol somehow I totally missed the fact that you write it with JS. That is a neat approach.

Hey, quick question: Is there a way to handle async behavior? Like in puppeteer we had something like waitFor helper to wait while loading the page. I couldn't find a similar thing in your readme. Or is this framework for only static pages?

At this time it works with server-side rendered / static pages only and does not execute any JavaScript of the website.

The data extraction function of your script also runs aside the crawler, not inside the page.