News for hackers. Stuff that matters.
Re:Reddit appears to be restoring edited/deleted comments by llm_nerd on Friday June 16, 2023 @07:55AM (#36355027)
Hanlon's razor.

Anyone who has used Reddit for any period of time (or basically any similar site, which includes FB, Twitter, and even HN), has seen data consistency issues during high load (like periods of activism). Shards die or fall out of quorum, work queues get purged, etc. That's the reality of systems with a lot of very low value data.

Eventually consistent, but also perpetually inconsistent.

Re:Reddit appears to be restoring edited/deleted comments by seanalltogether on Friday June 16, 2023 @08:26AM (#36355425)
Yeah people are going into overdrive right now with speculating all the bad things reddit is doing. Someone claimed that reddit admins had taken over their local city's subreddit to install new mods, as if that was a priority for them right now.
Re:Reddit appears to be restoring edited/deleted comments by goykasi on Friday June 16, 2023 @08:00AM (#36355100)
Yes, I have a strong love/hate relationship with multi-DC Cassandra.
Re:Reddit appears to be restoring edited/deleted comments by willis936 on Friday June 16, 2023 @08:08AM (#36355203)
Does Hanlon's razor apply when the actor has openly stated malicious intent?
Re:Reddit appears to be restoring edited/deleted comments by MichaelZuo on Friday June 16, 2023 @08:10AM (#36355235)
Which is a bit surprising since nowadays an entire reddit instance could fit on a single server rack.
Re:Reddit appears to be restoring edited/deleted comments by hospitalJail on Friday June 16, 2023 @08:44AM (#36355659)
Its not a shock to anyone who knows how cheap data is, but it just proves there is no true 'delete'. I never deleted my facebook profile/pictures because I didn't want to shine a spotlight on my account.

I care more about what a 'deleted' post says, than any other post in a thread. I imagine there is a good reason for that.

Re:Reddit appears to be restoring edited/deleted comments by 4BynHmBFUxBD on Friday June 16, 2023 @09:07AM (#36355929)
Same here. I also thought if I deleted my FB account, they would just maintain a “shadow profile” for me.
Re:Reddit appears to be restoring edited/deleted comments by pontiacbandit8 on Friday June 16, 2023 @08:55AM (#36355771)
Data consistency on comments n years old?
Re:Reddit appears to be restoring edited/deleted comments by llm_nerd on Friday June 16, 2023 @09:03AM (#36355857)
Unless someone has invented a data system that includes time travel as a feature, the original date of the comments is irrelevant.
Re:Reddit appears to be restoring edited/deleted comments by jacksnipe on Friday June 16, 2023 @09:18AM (#36356082)
Unlike every other data operation, deletion is heavily regulated by law.
Re:Reddit appears to be restoring edited/deleted comments by llm_nerd on Friday June 16, 2023 @02:05PM (#36361023)
Sure. I can't wait to see all the lawsuits and fines overwhelming Reddit, and any other service that have occasional data consistency issues. This is heavily regulated by law, of course.

Of course this isn't "heavily regulated". No law demands 100% consistency when you run some third party script during an outage. Clicking "delete" on a comment, or having a script do it, is not some GPDR enforced action.

Re:Reddit appears to be restoring edited/deleted comments by jacksnipe on Sunday June 18, 2023 @08:31AM (#36379655)
You are incredibly wrong about this. Consent for data processing is the meat & potatoes of both GDPR and the largest enforcement actions that have happened through it. The delete button revoked consent.
Re:Reddit appears to be restoring edited/deleted comments by aprentic on Friday June 16, 2023 @11:41AM (#36358509)
That would be a reasonable hypothesis if our information was limited to a few isolated inconsistency issues.

The more posts we see deleted the less likely it is that it's an accident. I just checked and found out that posts from an account that I scrubbed and deleted 5 years ago are popping back up. That stretches "eventual" pretty thin.

We also have a large amount of external information that suggests that reddit is not acting in good faith, much of it from Spez himself.

Between the two of those, malice is more likely than stupidity to be the motivating factor.

Re:Reddit appears to be restoring edited/deleted comments by crazygringo on Friday June 16, 2023 @11:44AM (#36358572)
That would easily explain comments coming back that were hours old or even days old, but it's extremely unlikely to explain comments coming back that were months or years old, as others in this thread have found.

No database shard would ever have been left that far out of date due to high load.

The only Hanlon's razor I can imagine here is somebody wrote a query to try to undelete something small (intending a single user or something) and forgot the WHERE clause and started to undelete everything. And didn't realize until the query had been executing for several minutes.

Yikes.

(I do actually think this is the most likely, not because of spez's motivations, but because there's zero benefit for Reddit here. It might make sense for them to force subreddits back to public, but undeleting old comments has no strategic value right now.)

Re:Reddit appears to be restoring edited/deleted comments by llm_nerd on Friday June 16, 2023 @12:31PM (#36359437)
as others in this thread have found.

People are going back to revisit things they think they deleted a long time ago to find it not deleted.

It's possible that those items were deleted, and then suddenly undeleted over the past few days as Reddit plies its nefarious plans. Far more likely, they were never deleted in the first place, or had the same temporal issue and reappeared a short while after deletion, but long long ago. The person thought it was deleted, but it wasn't for the same reason people are finding deletes not to stick now in some cases.

Re:Reddit appears to be restoring edited/deleted comments by fho on Friday June 16, 2023 @11:52AM (#36358713)
> (I do actually think this is the most likely, because there's zero benefit for Reddit here.)

The obvious benefit would be to artificially keep interaction rates high by "grave robbing" old comments and accounts.

"Nobody is ever going to notice, those accounts are long dead!"

Re:Reddit appears to be restoring edited/deleted comments by crazygringo on Friday June 16, 2023 @11:54AM (#36358763)
What interaction rates? For what ends?

The metrics you use to attract investment are things like monthly active users, and whether a comment was deleted after it was made doesn't affect that at all.

Re:Reddit appears to be restoring edited/deleted comments by arendjr on Friday June 16, 2023 @12:12PM (#36359085)
From what I gather, a lot of Reddit usage comes from Google referrals, similar to how StackOverflow gets a lot of its traffic. If people leaving Reddit delete or scramble their comments (which some are definitely doing), they also screw up Reddit’s referral value and Google ranking, so Reddit has quite some motivation for trying to restore those comments.
Re:Reddit appears to be restoring edited/deleted comments by fho on Friday June 16, 2023 @02:38PM (#36361612)
I was thinking pure "comments per second" numbers, ie how much users interact with reddit. No idea what's the right word for that.
Re:Reddit appears to be restoring edited/deleted comments by notacoward on Friday June 16, 2023 @11:45AM (#36358582)
I've worked on many eventually consistent systems, up to exascale at one of the companies you mention, and that explanation just seems extremely unlikely. In a system using tombstones, those should (and generally will) go the same place as the originals so "missing shard" doesn't work. Over that time scale there should then have been multiple rounds of compaction, including those forced by disk or server replacements, which would have also purged that deleted/overwritten data. Nobody at that scale can afford to let uncompacted garbage accumulate forever. Therefore, if not tombstones, they must be deliberately keeping old versions, but in that case the issue just moves to the index and doesn't really change much. The only way I can see this happening is if there was an unrecoverable data loss that required restoring from backup. Also, let's not forget that true deletion is sometimes required under various regulatory systems. I know that at least two of the companies you mention have systems specifically dedicated to satisfying those requirements.

It's still possible that this is innocent(ish), but at low enough probability that malice really does seem like a compelling alternative explanation. After all, spez has been caught doing something very similar before. There's proof of malicious nature, not just an assumption. When a component in a system has a certain known behavior or limit, you don't just discard that knowledge due to some handy rule of thumb.

Re:Reddit appears to be restoring edited/deleted comments by nebula8804 on Friday June 16, 2023 @02:27PM (#36361432)
Maybe I have this wrong but looking at academic torrents, the entire pushshift archive from 2005 to 2022 is 2TB compressed. Are they really operating at a massive scale that you are describing? Pushshift is supposed to have ingested all of Reddit's comment data no?

(Of course this is assuming pushsift has gotten the majority of comments)

Re:Reddit appears to be restoring edited/deleted comments by notacoward on Friday June 16, 2023 @03:07PM (#36362136)
Well, let me put it this way: if they're operating at such a small scale, they have no reason to be using any eventually consistent data store, so explanations about missing shards etc. still don't work. There are still other possibilities, as I and others have outlined, but not enough to dismiss the possibility that spez is just up to his old tricks.