I assume the reason for statements like that goes something like:

1. A single person can run a mastodon instance in their spare time. Spinning up some containers for the app, a background worker and a database is quite simple.

2. Modern devops tooling makes it fairly trivial to spin up 10k instances of a container instead of 1, by just altering a number in a k8s manifest somewhere.

3. Ergo, a single person equipped with modern tooling (and sufficient funding) could spin up any number of mastodon instances.

4. Twitter is just a big mastodon instance.

5. Now that keeping everything up is sorted, add another 99 devs for feature development and you are done.

Now this is obviously faulty logic because points 3 and 4 are very false, but they look reasonable enough at first glance.

That is a straw man. There are other perfectly valid interpretations and distributions of a statement that 100 could maintain it, like this:

15 database admins

10 linux sys admins

5 kubernetes specialists

10 windows tech support

25 front end developers

15 back end developers w/ Scala

10 machine learning experts

Whether that makeup could or couldn't do it is a different question, or whether it would be a different mix; all of that is up for debate, but the 1/99 ratio is just one very specific, extreme, and laughable mix for anyone who has supported a system of any real size.

How is that valid for creating and maintaining Twitter? 15 backend engineers are not even sufficient for any single facet of Twitter. Plus, there are other initiaves that Twitter takes on that are probably outside the scope of what you consider Twitter. Take a look at just their open source initiatives: https://opensource.twitter.dev/projects/. Go to, e.g., Finagle's Github page and look at the number of contributors and commits. While you may disagree that all of this is essential for creating and maintaining Twitter, there are problems that creep in when your platform supports billions of users that are not solved either correctly or at all by existing libraries and services; plus, open source projects serve as important recruiting collateral for engineers, especially for companies that do not have a set of widely used tools that entrench engineers (e.g. AWS, GC, Azure).

Creating a husk of an app that looks vaguely like Twitter and supports hundreds of users is a weekend project for anyone with a modicum of talent; building a platform that supports billions of users and is monetized well enough to support itself is an entirely different beast.

I love how you just extrapolate from a single server with a few thousand users to Twitter scale with 200+ million users.

As though you can just add more database or add more servers like it's a startup using the cloud.

But please by all means educate us on which database you would pick to run Twitter.

Did you read the last line where I mentioned it was faulty logic?

Flat files and replication would probably get you unreasonably far.

I don't know why you are being down-voted. Premateralized flat files pushed out to heavy-duty CDNs would get you very, very far. I would think that dealing with CSAM, death-threats, and other things that would get your platform into trouble would be the more difficult problem to manage. I could very easily see the technical parts of Twitter being run with a few hundred people. I don't know what else would be involved for legal, finance, marketing, sales, etc. But I doubt it is 8,000+ people.

Did you actually read the whole comment you replied to?

also, iirc a database based architecture was why the failwhale image was made. iirc the threw out databases and went with something more analogous to email with tweets represented as files on file systems. That was was a lonnnng time ago though, they may have reinvented the architecture a dozen times since then.

There are some major pieces missing from your analysis: legacy code and infrastructure, and lack of good documentation.

These can make for a massive hairball of complexity that can swell the number of people needed to support it.

This reminds me of a talk I once saw by a Netflix SRE, who showed a crazy convoluted mess of a diagram with thousands of crisscrossing lines going everywhere, and him screaming "No one understands Netflix!!!"

If you think Netflix is bad, you should see architecture diagrams for one of the major ad platforms; 30+ service sub-systems get encapsulated as a tiny block of the massive diagram. Anyone who thinks complex systems can be built and maintained by a skeleton crew hasn't worked on complex systems or hasn't been exposed to their full scope.

> 1. A single person can run a mastodon instance in their spare time. Spinning up some containers for the app, a background worker and a database is quite simple.

Yes, using open source code built and maintained by 732 contributors: https://github.com/mastodon/mastodon/graphs/contributors

I couldn’t give you details, but I do know Instagram had 13 employees when Facebook bought them. Is Twitter really two-orders of magnitude more difficult to run?

Well, let’s see. IG was acquired by FB in April 2012 and had somewhere above 10M users at the time. Around the same time, Twitter had around 140M accounts in US alone, nearly 500M worldwide. Do we want to continue the apples vs oranges comparison further. Happy to keep citing scale differences. :)

Source: Wikipedia pages for both

Do you also have a source showing that the number of engineers required scales linearly with number of users?

It in fact scales exponentially.

Internal sources only, but topics like regionalization, localization, bespoke caching implementations, hardware-level optimizations, content moderation, policy adherence (e.g. GDPR, CCPA), long-term monetization (especially if supporting advertisers as a direct customer), 3P support, public APIs, documentation, SRE (a fledgling product doesn't need 5 9s), analytics (internal and for advertisers and 3P partners) and for the most part, security, are non-exhaustive examples of things you can mostly ignore when your product supports 10M users that are unavoidable when your product supports half of the US, a good chunk of the rest of the world, and other large businesses that consume you at large business scale.

You listed a bunch of requirements without an argument that fulfilling those requirements actually entails an exponential growth in employees.

That's true, but also hardware and software hasn't stood still in the past decade.

I'd definitely like to hear more about the scale differences. So far at best, you've accounted for one order of magnitude. How do you explain the second?

> Well, let’s see. IG was acquired by FB in April 2012 and had somewhere above 10M users at the time. Around the same time, Twitter had around 140M accounts in US alone, nearly 500M worldwide.

So increasing your user base by one order of magnitude requires increasing the number of employees by more than two orders of magnitude? Rate of employee acquisition should probably never outpace rate of user acquisition, so I think that's a pretty clear sign that something was off.

Was instagram selling advertising to giant brands yet when they were bought? Sales staff balloon because they are really good at selling themselves to hiring staff and also because any brand that pulls in more than $10M in revenue expects to be treated like the only king of the world in pretty much every interaction and literally requires handlers, and the number of handlers required scales with the size of the brand.

You can potentially solve the handlers issue by being ruthlessly cutthroat. If they want hand-holding, they can hire a third party to manage advertising on Twitter. They probably already do in fact, so if you are a third-party, do your job and know your tools.

As for sales staff being good at selling themselves, agreed, so maybe Musk's ruthless firing spree will end up as a good thing. Maybe.

>You can potentially solve the handlers issue by being ruthlessly cutthroat

No, that just results in those businesses and brands leaving you, unless you can provide them a LARGE revenue stream that is impossible to get anywhere. A large brand will absolutely give up a little money just to spite you and your company for not treating them like god.

In my mind the issue is less “is it conceivable that a small team can run a bare bones version of the Twitter app with hundreds of millions of users” and more “can a small team manage a big distributed system that was designed to be managed by many different teams, with no handoff period”.

> I do know Instagram had 13 employees when Facebook bought them.

It was also pathetically unprofitable, and had serious problems with inappropiate child photos, gore etc.

Some of those problems require man power, there is no 10 man team who are very good at devops who can solve that.

The person I responded to was specifically talking about the tech stack. I wholeheartedly agree that the difficult part of running a service like Twitter lies in the soft problems.

But the profitability requieres a big tech team regarding ads and recomendation algos etc.

And moderation requieres humans but also a big tech team regarding bots, known offenders, etc.

it's not exclusively a tech problem but in a tech company tons of those responsabilities will be handled, addressed and solved by product and tech teams. And on 10 people, no matter how smart, they got no chance

Child safety advocates have praised Elon for his quick action in removing inappropiate child photos from Twitter, as well as dealing with the hashtags traffickers use. Something people have been asking for years for and little was being done.

Nobody knows what the secret hashtags are. The claim is unverified and unverifiable, and probably signal-boosted from a single wild tweet.

Unless you wish to ask for people to openly post child porn guidance you're gonna have to rely on the people actually following the issue for years. For what it's worth business insider claims they checked said hashtags and the content was gone[0]. Said user is allegedly going to release today an article in a corporate outlet that did fact check it[1] so we'll see.

I have no reason to distrust her and can't see any reason for her to lie about it.

0: https://www.businessinsider.com/elon-musk-says-child-sexual-... 1: https://twitter.com/elizableu/status/1595109108608544768

> Musk responded to the tweet saying that the issue is "Priority #1."

Obviously it's "top priority"! Is there any other acceptable answer? This answer means nothing except that he saw the question.

The reason for distrust is that the Q people have been insisting that Trump was secretly engaged in an enormous battle with pedophile rings, and using the typical, constant, and normal arrests of pedophile rings as proof. Now, another claim of a right-wing hero bravely picking up the sword and vanquishing the forces of evil, but again, no evidence! No proof at all. We just have to take her word for it, eh?

Of course, she can't give out the hashtags and allow independent verification, yet somehow hordes of pedophiles already know these hashtags? (How are they publishing those to each other, and if they have such a channel, why aren't they using that instead of Twitter?) So who is she keeping the hashtags from then?

So I remain super skeptical.

>Obviously it's "top priority"! Is there any other acceptable answer?

the answer means nothing, yes, the supposed good action that triggered the question is the important thing. The only relevant part of the article is them claiming to have verified it, so it's Business Insider's word added to hers.

I'm not even going to touch the conspiracy theory madness, it's irrelevant, and the person in question has zero signs of being afflicted by it, her entire existence in the platform seem to have been focused on actually working against the issue.

Its also been pretty widely publicized twitter's issue with CP[0][1][2][3], and given how hastags are the way you find things in the platform it's natural that's the way they'd do it.

> Of course, she can't give out the hashtags and allow independent verification, yet somehow hordes of pedophiles already know these hashtags?

She is giving it out to journalists and apparently they are confirming it, it's not a tragedy that someone does not want to amplify possible child abuse, actually teaching even more pedophiles how to find it.

> How are they publishing those to each other, and if they have such a channel, why aren't they using that instead of Twitter?

Twitter is protected by its sheer scale, is huge making detection harder than dedicated sites, and is far far more stable, safe and accessible than some darknet website they'd have to run themselves on average.

> So who is she keeping the hashtags from then?

From the general public to not further humilliate the victims, from other pedophiles because its not some hivemind, the list goes on.

Not everything is about Trump, and you're falling for the same level of conspiracy if behind all this out of all this you see pizzagate Qanon.

[0]: https://www.reuters.com/technology/exclusive-brands-blast-tw... [1]:https://nypost.com/2021/01/21/twitter-sued-for-allegedly-ref... [2]:https://www.theverge.com/23327809/twitter-onlyfans-child-sex... [3]:https://www.theblaze.com/op-ed/child-sex-abuser-twitter

I honestly don't understand how anyone reaches 2022, given all that we've experienced in the past several years, with enough faith in journalism remaining intact to pay any credence to "X happened, but we won't show you any evidence. Trust us."

For me, such a story is totally meaningless. It conveys no information about reality either way. The chances of truth or falsehood are exactly equal.

We have different epistemologies.

> from other pedophiles because its not some hivemind

Did you think critically about this? If so, for how long?

How are the pedophiles teaching the sooper-seekrit hashtags to each other? Is in the manual "So You've Decided to Become a Pedophile" that they send to new members of the vast conspiracy?

The problem with these conspiracy theories is that any inspection of how they might actually operate, day to day in the real world, is always neglected and handwaved away.

The whole point of a hashtag is that they spread virally, or are obvious terms. The very idea of a secret hashtag is an oxymoron.

When people say this it's because they assume there's not stupid ass politics or moat building.

THEY could probably do it with 100 people, YOU cannot.

100 people is most likely within the ballpark for a group of people whose sole purpose is to write and maintain twitter's tech stack. Unfortunately, that is not NEARLY the sole purpose of most people in businesses and that adds all kinds of productivity hits.

What happens is that people like yourself become convinced that's the only way to operate.

The question is not how many people to maintain an app with the functionality of Twitter, it's how many people to maintain Twitter. Twitter has to maintain Twitter's actual app, not an app that you built to be maintainable by 100 people. You can't say whether that's possible without knowing how Twitter actually works.

I could certainly build a version of twitter that didn't require so many people.

You could have saved Musk a lot of money then, too late now.