Ask HN: Best practices for ethical web scraping?

Re:Ask HN: Best practices for ethical web scraping? by pfarrell on Saturday April 04, 2020 @10:23AM (#22778365)

It won’t help you learn to write a scraper, but using the common crawl dataset will get you access to a crazy amount of data without paying to acquire it yourself.

https://commoncrawl.org/the-data/

[ Parent | on HN ]

Re:Ask HN: Best practices for ethical web scraping? by aspyct on Saturday April 04, 2020 @12:55PM (#22779509)

Cool, didn't know about this. Thanks!

[ Parent | on HN ]

Re:Ask HN: Best practices for ethical web scraping? by Reelin on Saturday April 04, 2020 @05:33PM (#22781719)

> As part of my learning in data science, I need/want to gather data.

Also not web scraping, but a few other public data set sources to check.

https://registry.opendata.aws

https://github.com/awesomedata/awesome-public-datasets

[ Parent | on HN ]

Re:Ask HN: Best practices for ethical web scraping? by aspyct on Saturday April 04, 2020 @06:16PM (#22781993)

Thanks!

[ Parent | on HN ]

Re:Ask HN: Best practices for ethical web scraping? by smcnally on Sunday April 05, 2020 @12:21AM (#22783715)

also https://www.reddit.com/r/datasets

[ Parent | on HN ]

Re:Ask HN: Best practices for ethical web scraping? by aspyct on Sunday April 05, 2020 @03:35AM (#22784318)

The comments that keep on giving :D

[ Parent | on HN ]

Re:Ask HN: Best practices for ethical web scraping? by analyticascent on Monday April 06, 2020 @05:58AM (#22792197)

I appreciate you starting this thread, these are great resources people are posting.

Common Crawl is the data set to master if someone wants to use the fruits of web scraping without actually doing the web scraping.

[ Parent | on HN ]

Re:Ask HN: Best practices for ethical web scraping? by my_green_book on Saturday April 04, 2020 @08:34PM (#22782697)

This is quite useful. Thanks a lot

[ Parent | on HN ]