News for hackers. Stuff that matters.
Re:Show HN: Property Trends Scraped from Zillow by Panther34543 on Tuesday August 02, 2022 @08:19PM (#32326101)
What is the legality of scraping those sites?
Re:Show HN: Property Trends Scraped from Zillow by tndibona on Tuesday August 02, 2022 @08:23PM (#32326135)
Afaik, US Supreme Court has ruled that it is not illegal to scrape public data for a similar case with LinkedIn. As for me being a pain for Zillows bandwidth - I’ve requested them umpteen times to allow me to pay for their API. They just simple ignore me. I’ve found a way around their captcha, in the long term, I have to obtain the data directly from the MLSes
Re:Show HN: Property Trends Scraped from Zillow by skanga on Friday August 05, 2022 @03:05PM (#32360506)
Would you be willing to share how you get around their captcha?
Re:Show HN: Property Trends Scraped from Zillow by wraptile on Wednesday August 03, 2022 @03:47AM (#32328787)
It's perfectly legal!

That being said, you still need to be resource-polite. A lot of people scrape Zillow through browser automation toolkits like Selenium, Puppeteer etc. because it's a JS heavy website and these tools are really bandwidth intensive. This could, in theory, get you in trouble for DDOS.

Instead, since Zillow is using Next.js for their backend so, you can actually retrieve the dataset for any page just by parsing the nextjs cache. This can be done by selecting data in the <script id="__NEXT_DATA__"> node which requires minimal resources from both sides. e.g. in python:

  import json
  import httpx
  from parsel import Selector
  
  response = httpx.get("https://www.zillow.com/b/1625-e-13th-st-brooklyn-ny-5YGKWY/")
  script_data = Selector(text=response.text).css('#__NEXT_DATA__').get()
  script_data = json.loads(script_data)
  # all of the property data is here, for example building details:
  print(script_data['building'])
I wrote a tutorial on this if you'd like to learn more: https://scrapfly.io/blog/how-to-scrape-zillow/#scraping-prop...
Re:Show HN: Property Trends Scraped from Zillow by skanga on Friday August 05, 2022 @04:50PM (#32361821)
Outstanding tip. Thank you VERY much.