It's a whole different ballgame to build on top of an existing complex system already in production that was made to satisfy the needs at the time it was built but it now needs to support other features, bug fixes and supporting existing features but at scale while having 50+ engineers not step on each other and not break each others code in the process. 4 friends in the basement will not achieve more than 50+ engineers in this scenario, even when considering the inefficiencies of the difficulty in communication that come along with so many minds working on the same thing.

I have worked on a 1000+ engineer project and another that was 500+, but I'm on the same boat as GP. Both of those didn't needed 50+, and the presence of the extra 950/450 caused several communication, organisational and architectural issues that became impossible to fix on the long term.

So I can definitely see where they're coming from.

I'm also admittedly extremely curious what (broadly) had 1000 (and 500) engineers dedicated to it, when arguably only 50 were needed. Abstractly speaking that sounds a lot like coordinational/planning micromanagement, where the manglement had final say on how much effort needed to be expended where instead of allowing engineering to own the resource allocation process :/

(Am I describing the patently impossible? Not yet had experience in these types of environments)

Yep, that's a fair assessment!

The 1000+ one was an ERP for mid-large businesses. They had 10 or so flagship products (all acquired) and wanted to consolidate it all into a single one. The failure was more on trying to join the 10 teams together (and including lots of field-only implementation consultants in the bunch), rather than picking a solid foundation that they already owned and handpicking what needed.

The 500+ was an online marketplace. They had that many people because that was a condition imposed by investors. People ended up owning parts of a screen, so something that was a "two-man in a sprint" ended up being a whole team. It was demoralising but I still like the company.

I don't think it's impossible to notice, but it's hard... you can ask during interviews about numbers of employees, what each one does, ask for examples of what each team does on a daily basis. Honestly 100, 500, 1000 people for a company is not really a lot, but 100, 500, 1000 for a single project is definitely a red flag for me now, and anyone trying to pull the "but think of the scale!!!" card is a bullshit artist.

> trying to join the 10 teams together

oh no

(insert https://webcomicname.com/ here)

> rather than picking a solid foundation that they already owned and handpicking what needed.

Mmmm.

I wonder if a close alternative (notwithstanding lack of context to optimally calibrate ideas off of) might have involved leaving all the engineers alone to compare notes for 6-12 months with the singular top-down goal of "decide what components and teams do what best." That could be interesting... but it leans very heavily on preexisting competence, initiative and proactivity (not to mention conflict resolution >:D), and is probably a bit spherical-cow...

> The 500+ was an online marketplace. They had that many people because that was a condition imposed by investors.

*Constructs getaway vehicle in spare time* AAAAAaaaaaa

Sad engineering face :<

> I don't think it's impossible to notice, but it's hard... you can ask during interviews about numbers of employees, what each one does, ask for examples of what each team does on a daily basis.

Noted. Thanks.

> Honestly 100, 500, 1000 people for a company is not really a lot, but 100, 500, 1000 for a single project is definitely a red flag for me now, and anyone trying to pull the "but think of the scale!!!" card is a bullshit artist.

That makes a lot of sense, and also filed away.

Also, I recently read this which resonates quite strongly with the economy-of-efficiency scale problem (which I totally agree with): https://rachelbythebay.com/w/2022/01/26/swcbbs/, and the update, https://rachelbythebay.com/w/2022/01/27/scale/

The biggest thing I've been able to correlate are command styles: imperative vs declarative.

I.e. is management used to telling engineering how to do the work? Or communicating a desired end result and letting engineering figure it out?

I think fundamentally this is correlated with bloat vs lean because the kind of organizations that hire headcount thoughtlessly inevitably attempt to manage the chaos by pulling back more control into the PM role. Which consequently leads to imperative command styles: my boss tells me what to do, I tell you, you do it.

The quintessential quote from a call at a bad job was a manager saying "We definitely don't want to deliver anything they didn't ask for." This after having to cobble together 3/4 of the spec during the project, because so much functionality was missed.

Or in interview question form posed to the interviewer: "Describe how you're told what to build for a new project." and "Describe the process if you identify a new feature during implementation and want to pitch it for inclusion."

There really is an art to scaling problems to humans so the individual work (across management and engineering) falls within the sweet spot of cognitive saturation. TIL yet another dimension that can go sideways.

The signal to noise ratio is very appreciated.

How do you bridge the two systems? You build an interim system. But customers want new features, so those features need to be done twice (bridge+new) if you're lucky, three times (existing+interim+new) if not. Could a smaller team of 10x engineers come in and do better? First off, thanks for insulting all of us, as if none of us are 10x-ers. But no. There's simply not enough hours in the day.

We've all heard of large IT projects that failed to land and said "of course". But we don't hear about the huge ones that do. And plenty of them do land, quite succesfully, with these 200+ person teams where I, as an SRE, don't know the code for the system I'm supporting.

None of this is visible from the outside.

There is huge difference between building a system that could theoretically be scaled up and actually scaling it up efficiently.

At small scales, it's really easy to build on the work of others and take things for granted without even knowing where the scaling limits are. For example, if I suddenly find I need to double my data storage capacity, I can drive to a store and come back with a trunk full of hard drives the same day. I can only do that because someone already build the hard drives, and someone stocked the nearby stores with them. If a hyperscaler needs to double their capacity, they need to plan it well in advance, allocating a substantial fraction of global hard drive manufacturing capacity. They can't just assume someone would have already built the hardware, much less have it in stock near where it's needed.