News for hackers. Stuff that matters.
Re:Why Twitter didn’t go down: From a real Twitter SRE by WJW on Tuesday November 22, 2022 @06:06AM (#33704457)
I assume the reason for statements like that goes something like:

1. A single person can run a mastodon instance in their spare time. Spinning up some containers for the app, a background worker and a database is quite simple.

2. Modern devops tooling makes it fairly trivial to spin up 10k instances of a container instead of 1, by just altering a number in a k8s manifest somewhere.

3. Ergo, a single person equipped with modern tooling (and sufficient funding) could spin up any number of mastodon instances.

4. Twitter is just a big mastodon instance.

5. Now that keeping everything up is sorted, add another 99 devs for feature development and you are done.

Now this is obviously faulty logic because points 3 and 4 are very false, but they look reasonable enough at first glance.

Re:Why Twitter didn’t go down: From a real Twitter SRE by brobdingnagians on Tuesday November 22, 2022 @06:31AM (#33704613)
That is a straw man. There are other perfectly valid interpretations and distributions of a statement that 100 could maintain it, like this:

15 database admins

10 linux sys admins

5 kubernetes specialists

10 windows tech support

25 front end developers

15 back end developers w/ Scala

10 machine learning experts

Whether that makeup could or couldn't do it is a different question, or whether it would be a different mix; all of that is up for debate, but the 1/99 ratio is just one very specific, extreme, and laughable mix for anyone who has supported a system of any real size.

Re:Why Twitter didn’t go down: From a real Twitter SRE by lightbendover on Tuesday November 22, 2022 @09:54AM (#33706452)
How is that valid for creating and maintaining Twitter? 15 backend engineers are not even sufficient for any single facet of Twitter. Plus, there are other initiaves that Twitter takes on that are probably outside the scope of what you consider Twitter. Take a look at just their open source initiatives: https://opensource.twitter.dev/projects/. Go to, e.g., Finagle's Github page and look at the number of contributors and commits. While you may disagree that all of this is essential for creating and maintaining Twitter, there are problems that creep in when your platform supports billions of users that are not solved either correctly or at all by existing libraries and services; plus, open source projects serve as important recruiting collateral for engineers, especially for companies that do not have a set of widely used tools that entrench engineers (e.g. AWS, GC, Azure).

Creating a husk of an app that looks vaguely like Twitter and supports hundreds of users is a weekend project for anyone with a modicum of talent; building a platform that supports billions of users and is monetized well enough to support itself is an entirely different beast.

Re:Why Twitter didn’t go down: From a real Twitter SRE by threeseed on Tuesday November 22, 2022 @06:46AM (#33704706)
I love how you just extrapolate from a single server with a few thousand users to Twitter scale with 200+ million users.

As though you can just add more database or add more servers like it's a startup using the cloud.

But please by all means educate us on which database you would pick to run Twitter.

Re:Why Twitter didn’t go down: From a real Twitter SRE by WJW on Tuesday November 22, 2022 @07:13AM (#33704917)
Did you read the last line where I mentioned it was faulty logic?
Re:Why Twitter didn’t go down: From a real Twitter SRE by withinboredom on Tuesday November 22, 2022 @07:09AM (#33704880)
Flat files and replication would probably get you unreasonably far.
Re:Why Twitter didn’t go down: From a real Twitter SRE by mhuffman on Tuesday November 22, 2022 @10:20AM (#33706710)
I don't know why you are being down-voted. Premateralized flat files pushed out to heavy-duty CDNs would get you very, very far. I would think that dealing with CSAM, death-threats, and other things that would get your platform into trouble would be the more difficult problem to manage. I could very easily see the technical parts of Twitter being run with a few hundred people. I don't know what else would be involved for legal, finance, marketing, sales, etc. But I doubt it is 8,000+ people.
Re:Why Twitter didn’t go down: From a real Twitter SRE by SadTrombone on Tuesday November 22, 2022 @08:01AM (#33705267)
Did you actually read the whole comment you replied to?
Re:Why Twitter didn’t go down: From a real Twitter SRE by chasd00 on Tuesday November 22, 2022 @10:25AM (#33706763)
also, iirc a database based architecture was why the failwhale image was made. iirc the threw out databases and went with something more analogous to email with tweets represented as files on file systems. That was was a lonnnng time ago though, they may have reinvented the architecture a dozen times since then.
Re:Why Twitter didn’t go down: From a real Twitter SRE by pmoriarty on Tuesday November 22, 2022 @07:47AM (#33705170)
There are some major pieces missing from your analysis: legacy code and infrastructure, and lack of good documentation.

These can make for a massive hairball of complexity that can swell the number of people needed to support it.

This reminds me of a talk I once saw by a Netflix SRE, who showed a crazy convoluted mess of a diagram with thousands of crisscrossing lines going everywhere, and him screaming "No one understands Netflix!!!"

Re:Why Twitter didn’t go down: From a real Twitter SRE by lightbendover on Tuesday November 22, 2022 @09:58AM (#33706497)
If you think Netflix is bad, you should see architecture diagrams for one of the major ad platforms; 30+ service sub-systems get encapsulated as a tiny block of the massive diagram. Anyone who thinks complex systems can be built and maintained by a skeleton crew hasn't worked on complex systems or hasn't been exposed to their full scope.
Re:Why Twitter didn’t go down: From a real Twitter SRE by KoftaBob on Tuesday November 22, 2022 @08:41AM (#33705631)
> 1. A single person can run a mastodon instance in their spare time. Spinning up some containers for the app, a background worker and a database is quite simple.

Yes, using open source code built and maintained by 732 contributors: https://github.com/mastodon/mastodon/graphs/contributors