Re:Show HN: Property Trends Scraped from Zillow
by Panther34543 on Tuesday August 02, 2022 @08:19PM
(#32326101)
What is the legality of scraping those sites?
[ Parent
| on HN ]
Re:Show HN: Property Trends Scraped from Zillow
by tndibona on Tuesday August 02, 2022 @08:23PM
(#32326135)
Afaik, US Supreme Court has ruled that it is not illegal to scrape public data for a similar case with LinkedIn. As for me being a pain for Zillows bandwidth - I’ve requested them umpteen times to allow me to pay for their API. They just simple ignore me. I’ve found a way around their captcha, in the long term, I have to obtain the data directly from the MLSes
Re:Show HN: Property Trends Scraped from Zillow
by skanga on Friday August 05, 2022 @03:05PM
(#32360506)
Would you be willing to share how you get around their captcha?
Re:Show HN: Property Trends Scraped from Zillow
by wraptile on Wednesday August 03, 2022 @03:47AM
(#32328787)
It's perfectly legal!
That being said, you still need to be resource-polite. A lot of people scrape Zillow through browser automation toolkits like Selenium, Puppeteer etc. because it's a JS heavy website and these tools are really bandwidth intensive. This could, in theory, get you in trouble for DDOS.
Instead, since Zillow is using Next.js for their backend so, you can actually retrieve the dataset for any page just by parsing the nextjs cache. This can be done by selecting data in the <script id="__NEXT_DATA__"> node which requires minimal resources from both sides. e.g. in python:
import json
import httpx
from parsel import Selector
response = httpx.get("https://www.zillow.com/b/1625-e-13th-st-brooklyn-ny-5YGKWY/")
script_data = Selector(text=response.text).css('#__NEXT_DATA__').get()
script_data = json.loads(script_data)
# all of the property data is here, for example building details:
print(script_data['building'])
I wrote a tutorial on this if you'd like to learn more: https://scrapfly.io/blog/how-to-scrape-zillow/#scraping-prop...