News for hackers. Stuff that matters.
Re:Web Scraping in Python – The Complete Guide by simonw on Tuesday February 20, 2024 @10:50AM (#39442740)
I strongly recommend adding Playwright to your set of tools for Python web scraping. It's by far the most powerful and best designed browser automation tool I've ever worked with.

I use it for my shot-scraper CLI tool: https://shot-scraper.datasette.io/ - which lets you scrape web pages directly from the command line by running JavaScript against pages to extract JSON data: https://shot-scraper.datasette.io/en/stable/javascript.html

Re:Web Scraping in Python – The Complete Guide by thrdbndndn on Tuesday February 20, 2024 @11:14AM (#39443071)
Kinda tangent, but Playwright's doc (specifically, the intro https://playwright.dev/python/docs/intro ) confuses me. It asks you to write a test and then run `pytest`, instead of just letting you to use the library directly (which exists, but is buried in the main text: https://playwright.dev/python/docs/library).

I understand that using Playwright in tests is probably the most common use case (it's even in their tagline) but ultimately the introduction section of a lib should be about the lib itself, not certain scenario to use it with a 3rd-party lib B (`pytest`). Especially when it may cause side effect (I wasn't "bitten" by it but surely was confusing: when I was learning it before, I created test_example.py as said in a minefield folder which has batch of other test_xxxx.py files. And running `pytest` causes all of them to run, and gives confusing outputs. And it's not obvious to me at all, since I've never used pytest before and this is not a documentation about pytest, so no additional context was given.)

> tagline

Re:Web Scraping in Python – The Complete Guide by simonw on Tuesday February 20, 2024 @11:22AM (#39443186)
Hah yeah that's confusing.

https://playwright.dev/python/docs/intro is actually the documentation for pytest-playwright - their pytest plugin.

https://playwright.dev/python/docs/library is the documentation for their automation library.

I just filed an issue pointing out that this is confusing. https://github.com/microsoft/playwright/issues/29579

Re:Web Scraping in Python – The Complete Guide by PaulHoule on Tuesday February 20, 2024 @12:21PM (#39443978)
Back in the day I used to use HTMLUnit

https://htmlunit.sourceforge.io/

to crawl Javascript-based sites from Java. I think it was originally intended for integration tests but it sure works well for webcrawlers.

I just wrote a Python-based webcrawler this weekend for a small set of sites that is connected to a bookmark manager (you bookmark a page, it crawls related pages, builds database records, copies images, etc.) and had a very easy time picking out relevant links, text and images w/ CSS selectors and beautifulsoup. This time I used a database to manage the frontier because the system is interactive (you add a new link and it ought to get crawled quickly) but for a long time my habit was writing crawlers that read the frontier for pass N from a text file which is one URL per line and then write the frontier for pass N+1 to another text file because this kind of crawler is not only simple to write but it doesn't get stuck in web traps.

I have a few of these systems that do very heterogenous processing of mostly scraped content and something think about setting up a celery server to break work up into tasks .

Re:Web Scraping in Python – The Complete Guide by 0xDEADFED5 on Tuesday February 20, 2024 @01:32PM (#39444915)
agreed, playwright is great. it even has device emulation profiles built in, so you can for instance use an iphone device with the right screen size/browser/metadata automatically
Re:Web Scraping in Python – The Complete Guide by BeetleB on Tuesday February 20, 2024 @11:01AM (#39442882)
I actually use your shot-scraper tool (coupled with Mozilla's Readability) to extract the main text of a site (to convert to audio and listen via a podcast player). I love it!

Some caveats though:

- It does fail on some sites. I think the value of scrapy is you get more fine grained control. Although I guess if you can use any JS with shot-scraper you could also get that fine grained control.

- It's slow and uses up a lot of CPU (because Playwright is slow and uses up a lot of CPU). I recently used shot-scraper to extract the text of about 90K sites (long story). Ran it on 22 cores, and the room got very hot. I suspect Scrapy would use an order of magnitude less power.

On the plus side, of course, is the fact that it actually executes JS, so you can get past a lot of JS walls.

Re:Web Scraping in Python – The Complete Guide by ravenstine on Tuesday February 20, 2024 @12:07PM (#39443787)
Readability is great, and I use it, but it's odd how half-assed the maintenance for it has been. I've haven't seen any noticeable improvements to it in quite some time, and when I've looked for alternatives, it usually turns out they're using it under the hood in some capacity.

Perhaps it's already being made obsolete by LLM technologies? I'd be curious to hear from anyone who's used a locally running LLM to extract written content, especially if it's been built specifically for that task.

Re:Web Scraping in Python – The Complete Guide by BeetleB on Tuesday February 20, 2024 @02:04PM (#39445425)
> Readability is great, and I use it, but it's odd how half-assed the maintenance for it has been. I've haven't seen any noticeable improvements to it in quite some time

What improvements are you looking for? For me, it works over 95% of the time, so I'm happy. Occasionally it excises a section (e.g. "too short" heuristic), and I wish it was smarter about it. But like you, I haven't found better alternatives. I also need something I can run in a script.

> Perhaps it's already being made obsolete by LLM technologies? I'd be curious to hear from anyone who's used a locally running LLM to extract written content, especially if it's been built specifically for that task.

It would be good to benchmark this across, say, 50 sites and see which one performs better. At the moment, I don't know if I'd trust an LLM more than Readability - especially for longer content. Also, I wouldn't use it to scrape 90K sites. Both slow and expensive!

Re:Web Scraping in Python – The Complete Guide by ravenstine on Tuesday February 20, 2024 @05:40PM (#39447942)
Although it works most of the time, I've found it's common for it to either pick up things that shouldn't be included or it only picks up something like the footer but not the actual body. This can be true even when, upon inspection, there's no clear reason why the body couldn't be identified. It's particularly problematic on many academic articles that are in HTML (sort of ironic). I'd also like a bit more normalization built in, even if it's turned off by default.
Re:Web Scraping in Python – The Complete Guide by simonw on Tuesday February 20, 2024 @11:05AM (#39442932)
Wow, you're really putting it through its paces!

When you ran it against 90,000 sites were you running the "shot-scraper" command 90,000 times? If so, my guess is that most of that CPU time is spent starting and stopping the process - shot-scraper wasn't designed for efficient start/stop times.

I wonder if that could be fixed? For the moment I'd suggest writing Playwright code for 90,000 site scraping directly in Python or JavaScript, to avoid that startup overhead.

Re:Web Scraping in Python – The Complete Guide by BeetleB on Tuesday February 20, 2024 @02:00PM (#39445366)
Yes, indeed I launched shot-scraper command 90K times. Because it's convenient :-)

I didn't realize starting/stopping was that expensive. I thought it was mostly the fact that you're practically running a whole browser engine (along with a JS engine).

If I do this again, I'll look into writing the playwright code directly (I've never used it).

Re:Web Scraping in Python – The Complete Guide by Oras on Tuesday February 20, 2024 @10:52AM (#39442768)
+1 for playwright. The codegen is a brilliant way to simplify scraping.
Re:Web Scraping in Python – The Complete Guide by sakisv on Tuesday February 20, 2024 @11:09AM (#39442993)
Came here to write about Playwright. I've been using it for the last ~13 months to scrape supermarket prices and it's been a great experience.
Re:Web Scraping in Python – The Complete Guide by mrtimo on Tuesday February 20, 2024 @11:51AM (#39443570)
would love to learn more about what you are doing with supermarket prices
Re:Web Scraping in Python – The Complete Guide by dommer on Tuesday February 20, 2024 @12:03PM (#39443715)
+1 Interested in this area as well.
Re:Web Scraping in Python – The Complete Guide by sakisv on Tuesday February 20, 2024 @05:36PM (#39447911)
My main drive was to document the crazy price hikes that's been going on in my home country, Greece, so I'm scraping its 3 biggest supermarkets and keep track of the prices of their products*

Had a lot of fun building and automating the scraping, especially in order to get around some bot catching rules that they have. For example one of them blocks all the requests originating from non-residential IPs, so I had to use tailscale to route the scraper's traffic through my home connection and take advantage of my ISP's CGNAT.

You can take a look here: https://pricewatcher.gr/en/

* I'm not doing any deduplication or price comparisons between the supermarkets, I only show historical prices of the same product, to showcase the changes.

Re:Web Scraping in Python – The Complete Guide by deanishe on Wednesday February 21, 2024 @08:11AM (#39453344)
I can't get prices on my local supermarket's website (in Germany) without selecting a specific branch.

Seems kinda suspicious to me.

Re:Web Scraping in Python – The Complete Guide by sakisv on Wednesday February 21, 2024 @10:33AM (#39455041)
Yeah in this case between these 3 supermarkets there are 2 options:

1. You choose your general location or 2. You don't choose

In the first option you get one less general category to choose from (for example they may not have fresh fish)

As far as I can tell, in both cases the supermarket closest to the delivery address is responsible for filling out your order and usually what happens is that they call you to let you know that they don't have something and suggest substitutions.

Re:Web Scraping in Python – The Complete Guide by samstave on Tuesday February 20, 2024 @12:02PM (#39443705)
I too choose this guys supermaket scraper!

What Ive long wanted was the the ability to map prices to SCUs by having folks simply take a pic of the UPC + price, just like gasbuddy or what not - in addition to scraping from grocery posting their coupon sheets online for scraping, in addition to people just scanning (non-PII) portions of receipts.

Can you share what you've made thus far?

* could it be used as an automated "price matching" finder? (for those companies that do a "we price match!" thing

Re:Web Scraping in Python – The Complete Guide by tnolet on Tuesday February 20, 2024 @12:29PM (#39444081)
100%. Playwright (which does have Python support) is completely owning this scene. The robustness is amazing.
Re:Web Scraping in Python – The Complete Guide by thundergolfer on Tuesday February 20, 2024 @12:31PM (#39444094)
We use shot-scraper internally to automate keeping screenshots in our documentation up-to-date. Thanks for the tool![1]

Agree that Playwright is great. It's super easy to run on Modal.[2]

1. https://modal.com/docs/guide/workspaces#dashboard

2. https://modal.com/docs/examples/web-scraper#a-simple-web-scr...

Re:Web Scraping in Python – The Complete Guide by 3abiton on Tuesday February 20, 2024 @02:09PM (#39445483)
How does it compare to selenium or puppeteer?
Re:Web Scraping in Python – The Complete Guide by black3r on Tuesday February 20, 2024 @03:19PM (#39446363)
Playwright is a rewrite of puppeteer by people who worked on puppeteer before, but now under Microsoft instead of Github. Not sure if it reached feature parity yet, but all the things we used to do with puppeteer work with playwright, and it seems to be more actively developed.
Re:Web Scraping in Python – The Complete Guide by sam2426679 on Tuesday February 20, 2024 @04:25PM (#39447119)
Ime playwright is selenium plus some, e.g. you can inspect network activity without having a separately configured proxy.
Re:Web Scraping in Python – The Complete Guide by sam2426679 on Tuesday February 20, 2024 @05:06PM (#39447621)
Can anyone recommend a good methodology for writing tests against a Playwright scraping project?

I have a relatively sophisticated scraping operation going, but I haven’t found a great way to test methods that are dependent on JavaScript interaction behind a login.

I’ve used Playwright’s har recording to great effect for writing tests that don’t require login, but I’ve found that har recording doesn’t get me there for post-login because the har playback keeps serving the content from pre-login (even though it includes the relevant assets from both pre and post login.)