News for hackers. Stuff that matters.
Re:Why Twitter didn’t go down: From a real Twitter SRE by fenomas on Monday November 21, 2022 @11:45PM (#33702029)
To me the analogy here is that the new boss rolled in and sold all the fire extinguishers. That by itself doesn't set the building on fire - it doesn't even increase the chances of a fire occurring on any given day. But when one does...
Re:Why Twitter didn’t go down: From a real Twitter SRE by libraryofbabel on Tuesday November 22, 2022 @12:03AM (#33702178)
Every SRE knows that the leading cause of outages by far is someone making a change to the system. Twitter isn’t shipping many new features right now or even doing much maintenance. But eventually they will have to.

So the analogy becomes, the new boss sold all the fire extinguishers and also placed a short temporary ban on cooking in the building. But eventually people are going to start turning on stoves again… and then…

Re:Why Twitter didn’t go down: From a real Twitter SRE by woodruffw on Tuesday November 22, 2022 @12:18AM (#33702275)
This is correct, but also: a sudden decline in maintenance is a kind of change in its own right. Even automated processes have humans in the loop and manual sign-offs; there's always some cronjob or short-lived certificate somewhere that a human was dutifully maintaining.

Those things aren't going to fail any sooner than they would have anyways, but they're going to fail a lot harder due to the loss of institutional knowledge.

Re:Why Twitter didn’t go down: From a real Twitter SRE by RijilV on Tuesday November 22, 2022 @12:55AM (#33702535)
Welp, their TLS cert expires second week of the new year. I really hope for them that’s an automated process.
Re:Why Twitter didn’t go down: From a real Twitter SRE by jandrese on Tuesday November 22, 2022 @12:20AM (#33702303)
Except that he at the same time demanded that people invent an entirely new dish by the end of the week, and now they are scrambling to try to figure it out. Already the DMCA auto-takedown bot is apparently broken and people are posting entire movies on Twitter. I would expect other peripheral systems to start breaking down as nobody is maintaining them even as other parts of the system are being changed.
Re:Why Twitter didn’t go down: From a real Twitter SRE by kaba0 on Tuesday November 22, 2022 @12:37AM (#33702413)
2FA as well
Re:Why Twitter didn’t go down: From a real Twitter SRE by bboygravity on Tuesday November 22, 2022 @12:26AM (#33702346)
oh noes! Not the DMCA auto-takedown bot!

Said no-one in the entire world except a hand-full of Hollywood studio owners.

That bot shouldn't have existed in the first place, but I know that that falls under "just world fallacy" and is a naive thought.

Re:Why Twitter didn’t go down: From a real Twitter SRE by woodruffw on Tuesday November 22, 2022 @12:30AM (#33702372)
I interpreted the GP's comment less as a moral claim ("the DMCA bot is good") and more as a claim that the DMCA bot's failure is a strong indicator of internal instability (given that it sits directly at the intersection between Twitter's profit interests and microservices architecture).

Put another way: being unable keep a little bot running, one that keeps an entire industry happy, doesn't bode well for other components of the service.

Re:Why Twitter didn’t go down: From a real Twitter SRE by edgyquant on Tuesday November 22, 2022 @01:07AM (#33702590)
No it’s proof that people will take anything and run with it. This bot likely had low priority and that’s all
Re:Why Twitter didn’t go down: From a real Twitter SRE by woodruffw on Tuesday November 22, 2022 @01:34AM (#33702756)
Run with what?

It seems self-evident that the bot was considered low priority, since it isn’t working anymore. But nobody is disputing that: they’re saying that the fact that it is low priority does not bode well.

Re:Why Twitter didn’t go down: From a real Twitter SRE by yourapostasy on Tuesday November 22, 2022 @01:48AM (#33702852)
> This bot likely had low priority and that’s all

If it was a prerequisite to land $100M ARR from all the media properties’ marketing budgets to advertise the multi-billion dollar pipelines of the movie and entertainment industry, that lil’ bot was the gate to $11,415 per hour of revenue at risk if its uptime failed to sufficiently please the attorneys and auditors from those customers.

Re:Why Twitter didn’t go down: From a real Twitter SRE by toast0 on Tuesday November 22, 2022 @12:34AM (#33702399)
I mean, does Twitter want to be a party to a copyright lawsuit? If not, following legitimate looking DMCA notifications (and legitimate looking DMCA counter-notifications) and responding to suponeas as necessary gets you an affirmative defense for copyright infringement.

You may not like it, but having a bot do that probably saves a lot of legal hassle.

Re:Why Twitter didn’t go down: From a real Twitter SRE by edgyquant on Tuesday November 22, 2022 @01:08AM (#33702599)
Content providers cannot be held liable for user generated content under section 230. Try again.
Re:Why Twitter didn’t go down: From a real Twitter SRE by woodruffw on Tuesday November 22, 2022 @01:32AM (#33702741)
Hm? Intellectual property is explicitly carved out of 230, and even if it wasn’t: it isn’t user generated. Content providers are regularly found liable for infringement on their platforms, especially when the plaintiff can demonstrate willful negligence (which in this case would include discontinuing a seemingly effective scanning system.)
Re:Why Twitter didn’t go down: From a real Twitter SRE by toast0 on Tuesday November 22, 2022 @01:18AM (#33702651)
§230(e)(2) says

> (2) No effect on intellectual property law

> Nothing in this section shall be construed to limit or expand any law pertaining to intellectual property.

If section 230 from the CDA of 1996 provided immunity from copyright claims, there would be no reason to include procedural requirements for processing claims in the DMCA of 1998.

Re:Why Twitter didn’t go down: From a real Twitter SRE by threeseed on Tuesday November 22, 2022 @02:17AM (#33703009)
Twitter is a global company.

Many jurisdictions take an even harsher line when it comes to being complicit in intellectual property abuse. We saw this famously with The Pirate Bay, Napster etc.

Re:Why Twitter didn’t go down: From a real Twitter SRE by fatneckbeardz on Tuesday November 22, 2022 @04:51AM (#33703950)
youtube literally been sued over this exact issue multiple times.
Re:Why Twitter didn’t go down: From a real Twitter SRE by kaba0 on Tuesday November 22, 2022 @12:38AM (#33702425)
Well, Disney won’t care why their copyrighted material is publicly available, noone likes this sort of copyright, but if Elon wants to avoid huge fines he better (make someone) fix it ASAP.
Re:Why Twitter didn’t go down: From a real Twitter SRE by CodeMage on Tuesday November 22, 2022 @12:36AM (#33702410)
I wonder whether this handful of Hollywood studio owners will have any influence on Twitter's revenue at all...
Re:Why Twitter didn’t go down: From a real Twitter SRE by emodendroket on Tuesday November 22, 2022 @12:30AM (#33702374)
Well it is exposing them to significant legal risk if they no longer comply, isn't it?
Re:Why Twitter didn’t go down: From a real Twitter SRE by edgyquant on Tuesday November 22, 2022 @01:06AM (#33702584)
Are we in support of DMCA now?
Re:Why Twitter didn’t go down: From a real Twitter SRE by thrwyoilarticle on Tuesday November 22, 2022 @07:08AM (#33704867)
I don't think you really think that's what OP is saying.
Re:Why Twitter didn’t go down: From a real Twitter SRE by afavour on Tuesday November 22, 2022 @12:22AM (#33702313)
> Twitter isn’t shipping many new features right now

Is that true? I thought one of Elon’s big pushes was launching the whole Blue Tick subscription thing. That doesn’t feel like a small feature.

Re:Why Twitter didn’t go down: From a real Twitter SRE by EdwardDiego on Tuesday November 22, 2022 @12:34AM (#33702400)
Given that Twitter already offered premium API access, they've got billing in place, so now they add a new form that, once your credit card is verified, flicks a boolean on an account that was previously flicked by another process.

It might not be small, but it's not exactly huge.

Re:Why Twitter didn’t go down: From a real Twitter SRE by fomine3 on Tuesday November 22, 2022 @12:37AM (#33702416)
Blue is paid by in app purchase that seems to haven't implemented until that.
Re:Why Twitter didn’t go down: From a real Twitter SRE by EdwardDiego on Tuesday November 22, 2022 @12:46AM (#33702480)
As I said, a new form.

They already had billing integration, even if you're accessing it via a new route, and they already had a boolean on your account.

Like I said, not small, but not exactly huge either.

Re:Why Twitter didn’t go down: From a real Twitter SRE by sundvor on Tuesday November 22, 2022 @01:02AM (#33702567)
Except there was a crapload of social engineering testing around this feature that was just skipped altogether.

Suddenly that simple change had the potential for catastrophic consequences.

Re:Why Twitter didn’t go down: From a real Twitter SRE by EdwardDiego on Tuesday November 22, 2022 @01:27AM (#33702720)
Yep, doing it right would've been harder, but Musk was after doing it right now.
Re:Why Twitter didn’t go down: From a real Twitter SRE by sundvor on Tuesday November 22, 2022 @08:16PM (#33713842)
I guess you could say doing it right wasn't exactly rocket science.
Re:Why Twitter didn’t go down: From a real Twitter SRE by EdwardDiego on Thursday November 24, 2022 @08:20AM (#33731000)
/golf clap.
Re:Why Twitter didn’t go down: From a real Twitter SRE by afavour on Tuesday November 22, 2022 @12:59AM (#33702547)
It’s more than that. For one it’s not an existing boolean, there are now two different kinds of blue tick that are presumably stored separately. Blue is also supposed to give the user fewer ads (while making them more relevant) as well as additional weight in feed ranking algorithms. It’s also intended to be offered worldwide which adds a lot of complication to things like payment flows.

I’m not saying it’s going to bring the site down tomorrow but that one feature touches on a lot of services.

Re:Why Twitter didn’t go down: From a real Twitter SRE by EdwardDiego on Tuesday November 22, 2022 @01:31AM (#33702737)
I imagine their premium API was also available worldwide.

Okay, so two booleans, and checks of those booleans in a couple of algorithms.

This still isn't a huge change, it's not completely new functionality. Once again, not saying it's small, but it definitely isn't huge.

Re:Why Twitter didn’t go down: From a real Twitter SRE by afavour on Tuesday November 22, 2022 @10:12AM (#33706635)
Clearly worldwide payments is an issue otherwise they’d have rolled it out worldwide day one, and they didn’t. There must be something holding that back.

Plus I really don’t think you can compare B2B payments for premium API access to end user payments. Not least because they aren’t going to be going the same route: a huge number of them will be via Apple or Google in-app purchasing. Ask anyone who works with those systems, it isn’t a quick plug and play job.

In general though, a new subscription tier, feed algorithm changes, UI changes… if these aren’t, what is a big change in your book?

Re:Why Twitter didn’t go down: From a real Twitter SRE by miked85 on Tuesday November 22, 2022 @12:43AM (#33702452)
> Twitter isn’t shipping many new features right now

When was the last major new feature? The site has always seemed pretty stagnant.

Re:Why Twitter didn’t go down: From a real Twitter SRE by emodendroket on Tuesday November 22, 2022 @12:58AM (#33702544)
They outright built me-too versions of both Clubhouse and IG Stories, and a lot of stuff trying to encourage people to be nice.
Re:Why Twitter didn’t go down: From a real Twitter SRE by Bukhmanizer on Tuesday November 22, 2022 @12:55AM (#33702534)
I was just about to make the same analogy. The chance that anything will break when there are no new commits is way lower
Re:Why Twitter didn’t go down: From a real Twitter SRE by jonas21 on Tuesday November 22, 2022 @12:25AM (#33702339)
Perhaps a closer analogy would be that the new boss rolled in and threw away 80% of the fire extinguishers.

Whether that will spell disaster when a fire occurs depends on whether the building had too many fire extinguishers to begin with and whether the boss can buy new, better fire extinguishers to replace some of the ones they got rid of before there's a fire.

Re:Why Twitter didn’t go down: From a real Twitter SRE by fenomas on Tuesday November 22, 2022 @12:38AM (#33702424)
> Perhaps a closer analogy would be that the new boss rolled in and threw away 80% of the fire extinguishers.

If we're deep-diving it'd be closer to say that he rolled in and sold 80% of the stuff, largely sight-unseen, and if a fire breaks out he'll find out how much of that stuff was fire extinguishers.

Re:Why Twitter didn’t go down: From a real Twitter SRE by wardedVibe on Monday November 21, 2022 @11:51PM (#33702076)
Definitely stealing this as the right way to frame the issue. What would normally be a small kitchen mistake turns into no longer having an apartment complex.
Re:Why Twitter didn’t go down: From a real Twitter SRE by cratermoon on Monday November 21, 2022 @11:52PM (#33702084)
He also sold all the smoke detectors.
Re:Why Twitter didn’t go down: From a real Twitter SRE by dcow on Tuesday November 22, 2022 @12:02AM (#33702174)
Credit where it’s due though: he did supply a spare sink.
Re:Why Twitter didn’t go down: From a real Twitter SRE by gnicholas on Tuesday November 22, 2022 @12:14AM (#33702253)
You should generally ignore sunk costs.
Re:Why Twitter didn’t go down: From a real Twitter SRE by inferiorhuman on Tuesday November 22, 2022 @12:06AM (#33702199)
He rolled in, sold most of the fire extinguishers, and made a big show of trying to make cherries jubilee while shoving his dick in one of the few remaining fire extinguishers. Let's be clear, it isn't just the erratic layoffs it's Musk's incessant meddling that's going to be Twitter's downfall. He literally took down SMS based 2FA because "microservices bad". He fired the payroll and tax departments (HR too?). He's scared off Twitter's main source of income while saddling it with significant debt.

As an SRE I would have been shocked if Twitter failed catastrophically (well moreso than broadly disabling authentication) in short order. However failure is pretty much inevitable at this point given the damage that E-Lon is actively doing.

Whatever. Twitter and Musk deserve each other.

Re:Why Twitter didn’t go down: From a real Twitter SRE by Andrew_nenakhov on Tuesday November 22, 2022 @12:12AM (#33702235)
> However failure is pretty much inevitable at this point

I'm pretty sure there will be no failure at all, and Twitter will work just fine.

Re:Why Twitter didn’t go down: From a real Twitter SRE by afavour on Tuesday November 22, 2022 @12:25AM (#33702334)
There have already been failures. If I recall the system for sending two factor auth codes via SMS was down for the best part of a day.

Not the end of the site by any means but cracks are showing.

Re:Why Twitter didn’t go down: From a real Twitter SRE by Andrew_nenakhov on Tuesday November 22, 2022 @03:50AM (#33703560)
Well, define 'failure'. Minor outages like the one you are talking about were happening from time to time long before Musk bought Twitter, and it even suffered long outages frequently - remember all those fail whales?

I meant that there will be no catastrophic failure that will permanently (or even for a few days) stop Twitter from working at all.

Re:Why Twitter didn’t go down: From a real Twitter SRE by fatneckbeardz on Tuesday November 22, 2022 @04:55AM (#33703987)
failure. noun. with no security team, hackers are able to get in easily.

everyone's DMs leak, all the anonymous accounts have their identity revealed, and all Twitter's clients (advertisers) have their bank account info made public.

Re:Why Twitter didn’t go down: From a real Twitter SRE by Andrew_nenakhov on Tuesday November 22, 2022 @05:26AM (#33704183)
Yeah, I don't think anything remotely close to it will happen, unless some of the fired developers have left themselves some backdoors which they'll give (sell) access to it to someone.
Re:Why Twitter didn’t go down: From a real Twitter SRE by fatneckbeardz on Tuesday November 22, 2022 @05:33AM (#33704231)
does this stuff happen automatically? is there a robot that goes out and reads about all the new zero-day exploits and patches all the software without human intervention?
Re:Why Twitter didn’t go down: From a real Twitter SRE by inferiorhuman on Tuesday November 22, 2022 @12:26AM (#33702350)
On Hacker News, everyone's a comedian! And, yeah, as the sibling comment pointed out there've already been failures as a result of some musky action. While Twitter isn't likely to fail on its own, E-Lon is actively causing problems. You need people to deal with that, and even if he had motivated, relevant, and competent engineers… how long will they stay motivated without a paycheck?

Let's not forget that whatever code monkeys are left are now personally liable for running afoul of the FTC. Whatever motivation they may have now will run out pretty damn quick once they stop getting paid.

Re:Why Twitter didn’t go down: From a real Twitter SRE by butlerm on Tuesday November 22, 2022 @12:46AM (#33702481)
Pretty sure that the developers that are left will not be liable for anything unless they are knowingly participating in criminal activity such as criminal negligence that is the direct cause of someone getting seriously injured or killed.

Generally speaking prosecutors want to target the highest level individuals responsible for directing such activity in the first place, not low level implementers who have little say one way or the other.

Re:Why Twitter didn’t go down: From a real Twitter SRE by inferiorhuman on Tuesday November 22, 2022 @01:09AM (#33702606)
For now I'm talking about the FTC consent decree, so administrative penalties not criminal charges. Musky fired the folks who were responsible for ensuring compliance.

https://techcrunch.com/2022/11/11/musks-lawyer-tells-twitter...

Re:Why Twitter didn’t go down: From a real Twitter SRE by Andrew_nenakhov on Tuesday November 22, 2022 @03:52AM (#33703576)
I think that he will soon bring in new developers/support engineers who wouldn't have questionable loyalties and grudges against the new management.
Re:Why Twitter didn’t go down: From a real Twitter SRE by inferiorhuman on Tuesday November 22, 2022 @07:30AM (#33705033)
How do you propose Musky does that with no payroll department? E-Lon walked back the mandatory return-to-office policy last week. Surely if he could find (or thought he could find) suitable replacements he would be pretty comfortable demanding RTO.
Re:Why Twitter didn’t go down: From a real Twitter SRE by Andrew_nenakhov on Tuesday November 22, 2022 @10:32AM (#33706832)
I'm not really familiar with how they do payrolls in USA, but I'm pretty sure it isn't some rocket science (pun intended) and can be done rather cheaply by an outsourced firm.

Also, why are you (and many others here) refer to Musk as "E-Lon"? Is it supposed to be a derogatory nickname?

Re:Why Twitter didn’t go down: From a real Twitter SRE by inferiorhuman on Tuesday November 22, 2022 @06:05PM (#33712817)
It's simply an abbreviated version of Elongated Muskrat as far as I'm concerned. Payroll is easily one of (if not the) most complex systems at any company. It's not just statutory stuff but personnel stuff as well. With Twitter you're not just dealing with 50 states and the feds, but with every other company in which Twitter has (had?) employees. There's a cottage industry of payroll firms precisely because payroll is so obscenely complex.

Even if you outsource it you'll still need people within your company to manage your service provider. At one company I worked for they got all of their outsourced HR+payroll for free (indefinitely) because the provider (Gevity) consistently fucked up everything they touched. This was at a company of like thirty people.

If you're suggesting Twitter can simply outsource payroll, sure. But you do that before you fire your whole payroll department. You still need people to handle the transition.

Re:Why Twitter didn’t go down: From a real Twitter SRE by quickthrower2 on Monday November 21, 2022 @11:58PM (#33702139)
My analogy.

Someone purchased some land for $1. Built a house for say $100. And now spends $100,000 a year making it the perfect place to rent, receiving $100,000 a year in rent.

Someone comes along and borrows $1m to buy that house. They feel ripped off but eventually are force to go ahead with the purchase. As a result they have to pay $100,000 a year in interest. They need this thing to be profitable!

To do this they need to cut back on the $100,000 a year spent. They decide go go in quickly and so email all the services saying "go hard or go home". So the plumbers, tradie, cleaners etc that don't like it leave.

As a side hustle also charge visitors to the house $9 to be allowed to wear their bowtie they used to wear for free.

Some of the people do maintenance jobs and improvements. They keep the termites out, fix subsidence issues, and so on.

And the house didn't fall down within 3 weeks of it being purchased.