How to Scrape Zillow with Python and Scrapy

Re:How to Scrape Zillow with Python and Scrapy by cute_boi on Sunday July 03, 2022 @08:46AM (#31967290)

I tried to scrape zillow in past (2019) and it was giving me captcha headache. Didn't author faced similar issues?

[ Parent | on HN ]

Re:How to Scrape Zillow with Python and Scrapy by slezyr on Sunday July 03, 2022 @11:47AM (#31968573)

> As of 2022, Zillow uses PerimeterX for scraping countermeasures. This can be bypassed by using ISP proxies from BrightData.

Why ask without reading the article?

[ Parent | on HN ]

Re:How to Scrape Zillow with Python and Scrapy by cute_boi on Monday July 04, 2022 @07:29AM (#31975880)

Well, I used luminati( Currently BrightData). And, it was still giving me captcha. In fact, it was giving me more captcha in BrightData. So, I am not sure.

[ Parent | on HN ]

Re:How to Scrape Zillow with Python and Scrapy by Kimcha on Monday July 04, 2022 @12:56PM (#31979188)

There are services that use humans to solve captchas for a very reasonable price, such as 2Captcha.

Some of the protections are better than others. For example, you may need to use a real browser with Java script support instead of just HTTP requests.

Browserless works well if you enable the stealth functions.

[ Parent | on HN ]