> move their infrastructure to AWS/Google Cloud which take the company to profitability

How exactly does that work?

Yeah, that was my one gripe with the comment. If you've already invested in creating your own infrastructure like Twitter has, moving to a public cloud is throwing money down the drain.

OTOH, continuing to develop that infrastructure may be continuing to place money on a losing bet... If they're actually running their own infra cheaper than cloud would allow, they should absolutely keep going with their own. But it's foolish to keep doing something simply because it's what you're already doing.

There's a fairly highly probability this is true?

AWS/GCE are not cheap. They're cost-effective for certain classes of use-case, but beyond a given scale, or with certain base-level requirements, you're way better off colocating a bunch of a machines.

My understanding is Netflix uses AWS for their infrastructure. Are they not on a similar scale as twitter with similar needs? Or is there a difference I am not seeing, perhaps a different use case. (This is a serious question, I am not doubting you.)

I believe that Netflix mainly uses AWS because their needs tend to surge dramatically at certain times, so that if they built their own worldwide server network, it would have to be vastly over-provisioned for when people get home at 7 PM and start streaming.

Plus they were a pretty early-adopter of AWS, and they have a ton of infrastructure running on it already.

AWS can be cheap if you architect right. However, bandwidth costs are going to kill you at AWS.

I'm intrigued that you're unfamiliar with the idea of a sunk cost.

You misunderstand my comment. Understandably so, I can see I wasn't as clear as I should have been. Let me clarify: Twitter has already spent a lot of money developing their own infrastructure. They now get to enjoy significantly cheaper ongoing infrastructure costs versus the cloud, as per the sibling comment. Moving to the cloud would be "throwing money down the drain" because they would have to invest significant resources into the migration and only have an increased monthly server bill as a result.

Generally, at any scale over ~500 computers that run continuously, investing in private infrastructure is going to be 1/2 to 1/5 the cost of public cloud, including people costs. The real levers for improving engineering spend are around complexity of your actual serving software, which doesn't depend on / isn't really influenced by who you buy your computers from.

I don't buy that claim at all. No way you're building private infrastructure for 1/5 the cost of Azure or AWS. Or even 1/2.

If you're comparing some servers in a 4th tier colo with a poor SLA to AWS and pretending that devops time is free, then sure. But if you're actually building something with comparable reliability and availability, then no way. You're going to spend a lot more than you'd expect on extra capacity, a team of service engineers, load balancers, multiple locations, etc.

What availability does Twitter actually deliver? I might be getting mixed up with reports about other sites or biased toward older events that happened when Twitter's architecture was less mature, but it seems like it's at least a semi-regular victim of "break the Internet" phenomena.

Twitter hasn't gone down in ages. (At least 2-3 full years.) It's reliable enough that hundreds of sites use Twitter as their "if our site is down, check for updates here" alternative.

> It's reliable enough that hundreds of sites use Twitter as their "if our site is down, check for updates here" alternative.

This has nothing to do with twitter's reliability. It just means companies have said twitter is more reliable than they are so look for our update on twitter opposed to some company specific system status page.

You're severely discounting the very large margin those companies take. It is not hard to build a medium sized infrastructure, with quality, cheaply - if you have access to expertise. This usually means hiring a few infrastructure jockeys from the larger companies.

Your SLA will likely be better than with public cloud.

You should absolutely build a public cloud competitor if you believe your numbers. By your numbers you'd make a minimum 100% margin and as much as 400%.

Unfortunately two things in the way of that are will and the cost of building a long tail of product features. First, that I'm not interested in running a standalone business, and second - my research here (it's been a lot) depends on being able to spend 20% effort on an 80% solution. I don't build a dynamo or a SNS and don't need a sales and marketing budget when I'm embedded in a large company building something purpose-built. The overhead of normalizing for industry wide generic use cases kills the 500% dream.

There's obviously room here though because AWS definitely takes a margin and that means opportunity!

Pretty sure they are already on AWS, also

They run their own data centers. https://blog.twitter.com/2016/the-infrastructure-behind-twit...

They're not, aside from some smaller acquisitions that were already on it.

50 people to maintain a site the size of Twitter?

Serious question (because I've never worked on a site that large) does the size of the site mean anything as to how many engineers are needed to maintain it? I would think it would be more a function of complexity?

There is a loose correlation, yes.

Security, abuse / harassment, fraud, community interaction, developer relations, 24/7 maintenance & ops, APIs, marketing & PR, legal, advertising, physical security / cleaning etc (whether outsourced or not it's a cost), secretarial, accounting, human resources & recruiting, sales, engineering, various management and so on.

The notion that you could run a $2.5 to $3 billion sales business with 100 people is a very bad joke. To deliver on that kind of sales level, you need more than a hundred people working in your sales organization alone. There are only a few types of businesses that can operate that thin at that size of sales, most of which are in the financial world.

Twitter doesn't need to cut down to 100 people to become nicely profitable, they need to cut down to 1,000 - 1,500 (while reducing infrastructure costs). Facebook delivers $1.5 million in sales per employee. Twitter's overall business is less complex and easier to operate than Facebook, they could deliver a higher ratio.

I don't have any experience with large-scale sites either, but here's my view:

It is a function of complexity, but complexity is often necessitated by scale. More users means more problems with scale, which leads to increased complexity to address the problems, which leads to a larger team needed to maintain the software.

That being said, 50 engineers seems enough to me for maintenance.

I doubt 50 people is enough to even field the various legal requests Twitter gets during a normal month.

Whether or not it's 50 people to field legal requests, it's certainly more than zero. And I'm sure that you don't need to look hard to find complaints that, even given current staffing levels, it's hard to reach a human who can deal with a hijacked account/harassment/etc. Scaling up from narrowly scoped technical support takes a lot more people.

Whatsapp only had 50 engineers even when they reached 900M users. Now, granted, Twitter may be a more complex system, but assuming they would just be maintaining it, they could be enough.

WhatsApp isn't real-time, and there's no reason for traffic 2 weeks ago to be any bigger than tomorrow.

However, there's a Presidential Debate tonight so you know Twitter's traffic is going to be off the charts.

WhatsApp isn't real time?

Twitter and WhatsApp are both Real Time. However, if they could save a lot money going from Real Time to Near Real Time, they should do it.

I don't know where you live, but people use Whatsapp groups a lot during and after an event.

As many mentioned, Whatsapp runs on 1/20th the number of Twitter employees.

It does have less features, but many of Twitter's are of dubious value.

Look at WhatsApp.

And see what? Mostly 1-to-1 events, no search, no broadcast, no history, not badgered with legal matters on any comparable scale, ...

I don't think it's comparable.

Legal would be outsourced, marketing same, spam control - yup, the same. That leaves developers and critical function staff. Twitter could easily work with 200 staff from 2 locations.

I'm curious: do you have experience running an organization like this, or are you speculating?

Working for. I was working for an outsourcing company that catered for companies like that. My hourly rate were ridiculously high, but still companies would be better off hiring us, than employing.

I agree that they should cut costs, but I don't agree with your thoughts of how they do so.

One of the notable things about Twitter is that they're a huge marketing machine. They have hundreds of employees that handle sales, marketing, advertising. Hell, they've acquired startups that handle advertising analytics, and have brought them in-house to build their own platform. They now directly sell contracts to TV channels and businesses that allow them to view how their brands interact on Twitter.

I'd argue that this has helped to create bloat. Anyone that has had to use Twitter's API over the years can see that they've gone from being an incredibly open company to building up a walled garden to protect their brand, in a similar way to how Facebook scaled back their efforts to be an application/gaming platform.

If I were Twitter, I'd give teams the opportunity to split from the company, but to continue to use their platform. If they believe in the product they've built within Twitter, they can be happy to compete against the open market. It allows them to drastically cut their workforce in key areas, while re-opening their product to the world. They can continue to be a sales company, but can fully allow others to continue to build great things on their platform, or to help shape how it works in the next decade.

But the question is how do you reduce? How do you decide who to keep, when whittling down to ~100 from 4000 (or whatever the current number is)?

From my experience, this type of re-org leaves the resultant company unsustainably top-heavy. I recently left a company who downsized but refused to hire anyone "junior level" because their pivot "requires all stars". And we all know, when everyone is a senior, no one is...