News for hackers. Stuff that matters.
Re:Show HN: Crawlee – Web scraping and browser automation li... by pqdbr on Tuesday August 23, 2022 @10:10AM (#32565174)
This looks really great. However, I can't find examples of how to handle scraping behind a login or a paywall, without having to 'type' credentials every time.
Re:Show HN: Crawlee – Web scraping and browser automation li... by pqdbr on Tuesday August 23, 2022 @10:16AM (#32565250)
Just found this on the Apify documentation: https://docs.apify.com/tutorials/log-into-a-website-using-pu...

Is there a similar guide for Crawlee?

Re:Show HN: Crawlee – Web scraping and browser automation li... by mnmkng on Tuesday August 23, 2022 @10:21AM (#32565328)
The example uses Crawlee already, you can just remove the

import { Actor } from 'apify';

and then all references to Actor and either remove them or replace them with Crawlee functions.

E.g. await Actor.openKeyValueStore() should be replaced with KeyValueStore.open()

It makes sense to add a separate example for Crawlee though. But it's true that it does not exist yet.

Re:Show HN: Crawlee – Web scraping and browser automation li... by mnmkng on Tuesday August 23, 2022 @10:18AM (#32565278)
You can use a headless browser (would recommend PlaywrightCrawler) to log in once and then use the session cookie until it expires in any crawler. When it expires, you can re-login and repeat the process.