A lot of commenters here on HN have already pointed that out already too. Unless they build a literal brick wall (paywall) around the site, that data can and will get scraped if the intention is to use for a model.
You could get it down to a science where you only scrape any new data whenever you train the next model.
[0]: https://old.reddit.com/r/reddit/comments/145bram/addressing_...
Reddit on mobile browser is a case study of insane dark patterns
Click to sort comments while not logged in? A popup appears asking to log in, with no close button. You have to click out of the box, but that’s not easily apparent
View an 18+ subreddit? Let them browse for 30 seconds, then ask they log in
Visit the site? Ask to login or use their app.
At this point, I’m more motivated to do anything but use their app if they’re this hellbent on getting me to download it.
The thing is, to me, reddit would never be a viable enterprise without the volunteers moderation. If reddit had to pay for that they’d never ipo. And if they think the moderators will stay and be exploited they’re wrong. Thus if they go through with this they’ll fail as a public company. There is no there where they going.
I'm convinced that this would have happened with or without OpenAI, especially with the mirage of an IPO on the horizon. Controlling the client to show ads and siphon data is just too valuable. Maybe the OpenAI thing pushed them to speed up the process.
How would that work from a legal perspective, though?
Let's say there's no paywall and Reddit's terms of use disallow unauthorized commercial use of their data. Wouldn't that be a violation of Reddit's terms and liable to some legal procedure?
It won't, with the LinkedIn vs HiQ precedent.
I think there are probably issues to address with scraping it blindly:
- Can Reddit imprint its data somehow? A watermark?
- Can Reddit prove that certain type of information appeared on Reddit first and thus that serves as proof its data was used without authorization?
If OpenAI can't work around this, I'm not sure they would be willing to cross any lines in terms of copyright, they've already done it with ChatGPT and I am guessing rules are only going to get stricter on this topic.
To me this feels like its opening up the door for the elimination of copyright as any algorithmic layer interjected between scrapped data and end users could claim to be "inspired".
Does Google have a special agreement with Reddit (and all other sites?) or is it legally "fair use" to reproduce web pages that are available freely online?
The only question is copyright, but I find it hard to argue that LLM training is not sufficiently transformative in 99% of cases.
The irony if lowtax and :10bux: ends up being right all along about how to build a community...
- When they started, I believe they boasted creating fake accounts to mimic engagement to help grow community
- It is painfully obvious the amount of political, gov sponsored, and corporate astro turfing that “gets through”. Both human and bot comment farms are real and have no doubt been artificially bolstering ideas and content for years
As I see it…
They have enabled this / looked the other away at this behavior forever in exchange for engagement. The super high tech AI chat bots are probably going to be welcomed for their clever content (hence why CEO could care less about its users anymore).
Their real value has always been having a controlling voice and being able to push viral ideas. The data thing is all hype noise IMO. It was never going to be a serious part of their IPO. Big messy gross company.
IMO, the future will be using LLMs with live search results - which of course will probably require a funding model better than the terrible Display Ads setup we have for such content now. So best of luck to Reddit on that front - I hope your ipo fails
N of 1, but I vastly prefer the Google Generative Results over chatGPT. Quality may not always be the same, but a “chat” seems like an awkward metaphor for finding info, and of course, chat GPT has no links to content when I’m worried about hallucinations.
I’m true google-search fashion, GenSearch has a lot less deep-in-the-weeds technical answers and will push you to simpler results. Eg. If you want to know what chemicals have a similar absorption properties to methane… you’re better off with ChatGPT or traditional search.
Main reason such sites came into existence was there was too much info on the Internet.
These sites where attempts at simplifying the numerous websites and blogs ppl had to manually discover and track.
They have meandered around that problem, and got totally distracted by all kinds of other problems (many self created).
Sure it will - asymptotically.
But it took 18 years to grow Reddit. Even if a solid alternative emerges in a fraction of that time -- that's still a multi-year gap. Plus we're at a significant knowledge deficit (for us lowly humans, never mind LLMs) if Reddit's archives don't re-emerge sometime soon.
(Google search has been braindead for some time now -- so we can leave that source out of the equation).
SO had publicly available, no-auth-required data dumps. This makes it difficult for them to know who is using their data. However, this surely isn't the case for Reddit who offered only API endpoints for this content, and I'm guessing you couldn't use .json to get the whole site (rate limits, etc). I wouldn't be inclined to believe that Reddit would miss a new major API user.
This is purely speculation disregarding Hanlon's razor, but I'm thinking that the API pricing comes down to killing two birds with one stone.
* Sama got to train his LLM on Reddit and some best-of-the-internet content there such as r/bodyweightfitness, informed discussions on niche topics etc for free. The catch-up players face prohibitive pricing.
* Third-party apps get killed, bringing the UX to Reddit's control. I think this is more important to Reddit than ad revenue, as they could've simply built an SDK for probably less than this PR nightmare will cost us.
* Their new development platform, however, hints at the Reddit app supporting serving "redditor-made apps" which "can be seamlessly reused between communities".
The description (and the idea to have apps in your app) weirdly reminds me of WeChat apps, and given that Tencent is a major shareholder in Reddit, I would consider the possibility that apps are something they're pushing. That idea has no chance of success without the UX being completely in Reddit's hands, even then it's questionable how it would work on Reddit.
Spez couldn't actually be that detached from Reddit?
Snoop Dogg owns more of reddit than Tencent. The Tencent investment is very overblown.
Basic math days Snoop owns twice as much.
Further, you could use the Reddit API to injest the full firehose of all site data in real time without violating rate limits. This is [one of the ways] how Pushift made their datasets.
Reddit was a lot more open that any other site approaching their size!
Who is "us"?
- next step is to stop third-party apps that generated these data
- then they let the moderators show their power
I’m not sure if people care about a CEO being exposed as a liar nowadays, but maybe some former Yahoo managers have another idea on how to destroy more value.
Specifically the ones that ended up in charge of tumblr, so they can suggest "ban adult content on a platform famous for its adult content"
I'm not so sure that's the case, for two reasons:
1) I have not stopped appending "reddit" to my searches or stopped visiting StackOverflow or other Stack* sites. ChatGPT is simply now an additional tool, and while there's some overlap in use cases there's also plenty I can do with *GPT that I couldn't with other sites.
2) To the extent that a training corpus relies on these (or any other) data sources, OpenAI has just made them much much more valuable! It's sort of like a mining company discovering that someone has an extremely valuable use case for decades of the mining company's tailings. This "someone" may have been allowed to haul away some of it without payment to the mining company, but now they know its value, now they can build a very lucrative business selling access to what remains, and what will be generates in the future.
That's all highly simplified though. It's of course much more complex than this, much more complex than can be captured in an HN discussion, but we can explore the outlines a bit, and even disagreement will reveal more and more of it.*
- The entire corpus of data the community had curated over the last XX years
- The "goodwill" that these platforms had developed towards third party developers in allowing developers to work with their data
- Potentially large amounts of traffic that would normally come to their sites via Google (e.g. site:reddit.com), that is now available instantly (and customized) via ChatGPT
Despite Reddit's probably closer connections to OpenAI than other startups through Y-Combinator and Sam Altman, I wonder how keen they are to actually work with a company that potentially destroyed a ton of their value, right before they were ready to IPO.