I scraped all of OpenAI's Community Forum

Re:I scraped all of OpenAI's Community Forum by dorkwood on Thursday March 28, 2024 @12:17PM (#39853538)

I did a bit of data scraping for fun in the past, but I was never quite sure of the legality of what I was doing. What if I was breaking some law in some jurisdiction of some country? Was someone going to track me down and punish me?

OpenAI has taught me that no one gives a shit. Scrape the entire internet if you want, and use the data for whatever you feel like.

[ Parent | on HN ]

Re:I scraped all of OpenAI's Community Forum by alt-glitch on Thursday March 28, 2024 @01:01PM (#39854353)

We were really heading someplace with The Semantic Web aka The Real Web 3.0 [1]

Alas we have to fight against the machines in order to properly read the internet thru machines.

I believe Discourse knowingly keeps its data easy to scrape though, so kudos to them!

[1]: https://en.wikipedia.org/wiki/Semantic_Web

[ Parent | on HN ]

Re:I scraped all of OpenAI's Community Forum by ifyoubuildit on Thursday March 28, 2024 @12:36PM (#39853959)

Do you think it would be better if someone did track you down and punish you? Which world do you want to live in?

[ Parent | on HN ]

Re:I scraped all of OpenAI's Community Forum by n0sleep on Thursday March 28, 2024 @12:43PM (#39854060)

I think large companies should be punished for stealing from people to make themselves richer.

[ Parent | on HN ]

Re:I scraped all of OpenAI's Community Forum by EcommerceFlow on Thursday March 28, 2024 @12:58PM (#39854320)

A precursor to this would have been that Linkedin lawsuit Microsoft lost, allowing that one company to scrape all of Linkedin (technically "public information").

[ Parent | on HN ]

Re:I scraped all of OpenAI's Community Forum by htrp on Friday March 29, 2024 @12:13PM (#39865772)

hiQ Labs v. LinkedIn

[ Parent | on HN ]

Re:I scraped all of OpenAI's Community Forum by bsuvc on Thursday March 28, 2024 @01:05PM (#39854423)

> OpenAI has taught me that no one gives a shit. Scrape the entire internet if you want, and use the data for whatever you feel like.

Cloudflare gives a shit.

My household had to use our 5G internet for most things for a week or two until our IP reputation recovered.

[ Parent | on HN ]

Re:I scraped all of OpenAI's Community Forum by stoorafa on Thursday March 28, 2024 @01:19PM (#39854628)

Yeah it’s probably worth renting a server if there’s any doubt about whether it’s wholly appropriate to do something

[ Parent | on HN ]

Re:I scraped all of OpenAI's Community Forum by htrp on Friday March 29, 2024 @01:07PM (#39866409)

Some sites just block the entire AWS/GCP ip address range.

[ Parent | on HN ]