News for hackers. Stuff that matters.

A comment on Ask HN: Would an easier way to scrape 100s of websites be useful to you?.

Re:Ask HN: Would an easier way to scrape 100s of websites be... by DamonHD on Friday September 22, 2023 @04:09PM (#37617102)
Scraping hundreds of site seems mostly unethical.

My sites are periodically on the wrong end of scrapers, greedy by design or in error, occasionally needing to be manually blocked or even legals threatened.

Just because something can be done, done't mean it should. It also doesn't mean that you should make it easier. Any more than offering 'better' SPAM engines...

[ Parent | on HN ]
Re:Ask HN: Would an easier way to scrape 100s of websites be... by asim-shrestha on Friday September 22, 2023 @04:31PM (#37617371)
Appreciate the input Damon, ethical concerns are definitely a consideration and we'll want to be respectful of mechanisms like robots.txt
[ Parent | on HN ]
Re:Ask HN: Would an easier way to scrape 100s of websites be... by webappguy on Thursday October 05, 2023 @11:39PM (#37786903)
Yawn, ignore that input. Scrape away!
[ Parent | on HN ]