News for hackers. Stuff that matters.

A comment on Ask HN: Would an easier way to scrape 100s of websites be useful to you?.

Re:Ask HN: Would an easier way to scrape 100s of websites be... by PaulHoule on Friday September 22, 2023 @04:14PM (#37617169)
My take is that the "idea people" usually underestimate how easy it is make user interfaces and overestimate the difficulty of scraping, API "integrations", etc. (It's what makes Zapier such a successful racket)
[ Parent | on HN ]
Re:Ask HN: Would an easier way to scrape 100s of websites be... by asim-shrestha on Friday September 22, 2023 @04:26PM (#37617310)
Our take is scraping a website in isolation is typically quite easy for a somewhat technical person.

Scraping at larger scale is where it becomes challenging, the problems we want to tackle are: 1. You need at least a bit of technical expertise to do things like configure selectors properly 2. Websites typically have moderation in place to block scrapers 3. Scrapers are prone to changes in the site layout 4. Creating on the order of 100s of scrapers is difficult and time consuming. Creating this many will amplify the previous issue

[ Parent | on HN ]