News for hackers. Stuff that matters.
Re:Web Scraping in Python – The Complete Guide by mrtimo on Tuesday February 20, 2024 @11:51AM (#39443570)
would love to learn more about what you are doing with supermarket prices
Re:Web Scraping in Python – The Complete Guide by dommer on Tuesday February 20, 2024 @12:03PM (#39443715)
+1 Interested in this area as well.
Re:Web Scraping in Python – The Complete Guide by sakisv on Tuesday February 20, 2024 @05:36PM (#39447911)
My main drive was to document the crazy price hikes that's been going on in my home country, Greece, so I'm scraping its 3 biggest supermarkets and keep track of the prices of their products*

Had a lot of fun building and automating the scraping, especially in order to get around some bot catching rules that they have. For example one of them blocks all the requests originating from non-residential IPs, so I had to use tailscale to route the scraper's traffic through my home connection and take advantage of my ISP's CGNAT.

You can take a look here: https://pricewatcher.gr/en/

* I'm not doing any deduplication or price comparisons between the supermarkets, I only show historical prices of the same product, to showcase the changes.

Re:Web Scraping in Python – The Complete Guide by deanishe on Wednesday February 21, 2024 @08:11AM (#39453344)
I can't get prices on my local supermarket's website (in Germany) without selecting a specific branch.

Seems kinda suspicious to me.

Re:Web Scraping in Python – The Complete Guide by sakisv on Wednesday February 21, 2024 @10:33AM (#39455041)
Yeah in this case between these 3 supermarkets there are 2 options:

1. You choose your general location or 2. You don't choose

In the first option you get one less general category to choose from (for example they may not have fresh fish)

As far as I can tell, in both cases the supermarket closest to the delivery address is responsible for filling out your order and usually what happens is that they call you to let you know that they don't have something and suggest substitutions.