Show HN: Crawlee for Python – a web scraping and browser automation library

Re:Show HN: Crawlee for Python – a web scraping and browser ... by manishsharan on Tuesday July 09, 2024 @01:16PM (#40918487)

Can this work on intranet sites like sharepoint or confluence , which require employee SSO ?

I was trying to build a small Langchain based RAG based on internal documents but getting the documents from sharepoint/confluence (we have both) is very painful.

[ Parent | on HN ]

Re:Show HN: Crawlee for Python – a web scraping and browser ... by jancurn on Tuesday July 09, 2024 @04:13PM (#40920588)

For this use case, you might use this ready-made Actor: https://apify.com/apify/website-content-crawler

[ Parent | on HN ]

Re:Show HN: Crawlee for Python – a web scraping and browser ... by mnmkng on Tuesday July 09, 2024 @04:06PM (#40920508)

Technically it can. You can log in with the PlaywrightCrawler class without issue. The question is if there’s 2FA as well and how that’s handled. Crawlee does not have any abstraction for handling 2FA as it depends a lot on what verification options are supported on the SSO side. So that part would need a custom implementation within Crawlee.

[ Parent | on HN ]