but it looks like somebody had their eyes on that issue
Expires: Tuesday, November 14, 2023 at 3:59:59 PM Pacific Standard Time
I’d take that bet.
Not everything is automated. There's a finite amount of time to create new automation, and that time goes first toward the things that happen hourly or daily. The things that happen monthly still get handled by humans. Not everything is in the runbook either, and what is often ends up scattered among many wiki pages and help notes in tools, so it's not easy for a newcomer to find. So those monthly things have to be done by people who remember.
They remember how to clean stuff up when a quota/capacity limit is being approached, and who to call when they need more, and they can expect a response. They remember how to recognize when their service is approaching overload, or about to enter an oscillating state, and they know how to nudge it back toward sanity before the errors start to pile up. None of that happens when whole groups are gone.
Sooner or later, a service will start to run off the rails in one way or another, and the person who inherited that service will either not recognize it or not find the solution in time. Then it's Fail Whale time.
Edit: downvoting doesn’t change reality
Pretty sure a number of AWS and Azure engineers believed the same thing.
Of course, even if it was automated, you still have to pay the bills, and if payroll quit as has been suggested in the thread, I'd be suprised if accounts payable is sticking around
The new one isn’t even indexed on crt.sh yet.
$ openssl s_client -servername twitter.com -connect twitter.com:443 2>/dev/null | openssl x509 -noout -dates
notBefore=Nov 14 00:00:00 2022 GMT
notAfter=Nov 14 23:59:59 2023 GMTNo, the way I think a frozen codebase will rear its head is when things that haven't been restarted in too long start to misbehave. When new updates are being pushed out frequently by a busy team, every machine (container, process, whatever) is being restarted regularly. People don't even realize what they're avoiding each time. Resource leaks, deadlocks, and timer issues can remain undetected a long time in that environment, then surface when the longevity increases. Even worse, some things tend to fail after a predictable amount of time, i.e. across most or all machines at once in a cluster.
I first saw this in 1990 at Encore, when our shiny new SMP version of UNIX became capable of staying up for a whole month at a time. Because of problems like those mentioned above, getting from one month to two took almost as long as getting from zero to one. I've seen the same scenario play out on many other systems since, and there's no reason to believe the systems within Twitter are immune.
Stasis is not the same as stability.
They developed those methods of dealing with JVM and scheduler issues because those problems rose in prominence against the background of a zillion other problems. They became the longest poles, most worth spending time and effort on. But those other problems still exist, with more appearing every day. Yes, even without code change, because environments and workloads change too. Anyone who has actually been responsible for large complex systems, not just in the sense of working at the same team/company and riding the coat-tails of those who actually understood it, knows that they require continued attention and always will. Entropy exists.
I'm pretty sure that that's the plan Elon Musk is executing. It's simple, brutal, nasty, and probably not very well thought through. And yes he's not being very nice about it (to put it mildly). But it's not completely crazy if you can get over how obnoxious he's being.
He's clearly not interested in keeping the current team. He's never going to win them over at this point and from his point of view they were part of the problem, not the solution in any case. He bought a company that was basically not profitable, bleeding cash, and fresh out of ideas on how to fix any of that for many years. And he's now on the spot to keep that train wreck going. So, first order of business is to just cut the cost. That means decimating the team to a fraction of its size.
The Twitter team is basically rage quitting at this point so that's solved. Again, I don't like how he's doing it but undeniably, there are going to be vastly less people on the payroll very soon and cost will be a lot lower. The law suits that follow will be interesting though. But my guess is that he'll just buy those off and that will be it.
Of those that remain, he can cherry pick a team to salvage/build whatever he needs and then re-enforce it with some outsiders. Building a micro blogging platform is not rocket science and Elon Musk is actually deeply into rocket science. So, I actually get his gut feeling that this simply should not require thousands of engineers obsessing about all sorts of arcane software bits and pieces that somehow implement Twitter. It's a micro blogging platform with a fairly simplistic feature set. And he just called bullshit on how that was done at Twitter. And we're talking about a person with a long track record of having absolutely no tolerance for bull shit.
Again, not liking his methods but I can see his line of reasoning here. He has no respect for the Twitter team and he actually wants them gone. He wants to rebuild Twitter in a way that is cost effective and interesting to him. And he is not a risk averse person. So, in his mind that means a small team that can execute fast. I kind of agree with that part.
Will this work? I don't know. I see no technical reason that this plan could not work. But the big question is if the Twitter brand name will be worth anything by the time he's done or if he's just going to shut down the whole thing and walk away from it. He can certainly afford to do that. But letting this thing coast along just never was going to be a thing.
Yeah, you can do Twitter on a weekend… if you only use it yourself and a few friends.
But we are talking about a platform with millions of users, all of which want real-time access to tweets written from anywhere in the world, including image and video content. Just getting to “no outages” and “low enough latency” parts is going to take a lot of effort to build from scratch. Don’t underestimate the complexity of Twitter.
I bet mastodon doesn't have thousands of very active committers. They are taking a lot of users right now. With a hacked together pile of ruby and javascript. Easy to forget that that is how Twitter once started out as well. The bloated org chart came later.
The scale of Twitter is large but not enormous. Basically a few trillions of messages overall. Most of them pretty small. All of them immutable (well so far). And lots of images/videos with some CDN in front of that. It's a lot of bytes but not a whole lot of complexity. And a couple hundred million users and user profiles. I can think of a quite a few ways of dealing with all that.
What's hard is things like algorithms and good search. But that's a space where you can achieve a lot with a small and focused team with access to lots of hardware. And obviously this is also something where Twitter has maybe been a bit underwhelming and struggling. User growth and engagement stagnated years ago for them. This was not a healthy social network. It's not like they were nailing this. Throwing more people at the problem wasn't working.
But also, how much time did Whatsapp have to get to that point? I'm not saying it's impossible, but that it's difficult and takes a lot of effort that shouldn't be underestimated.
> They are taking a lot of users right now.
And still less activity than Twitter, and with federation, and with some servers already locking new account creation (mastodon.social, the biggest one, doesn't let you create new accounts).
Again, it's not that it's impossible. It's that it's hard.
Lol
You did warn us it was crazy