So it's not always "those architecture astronaut guys used the company's resources to study their webscale fantasies and I came to save the day", but rather, "they did as well as they could using the choices and resources they stuck with and I happened upon the project with 20/20 hindsight".
Where I think I part ways with you is with this "they did as well as they could using the choices and resources they stuck with and I happened upon the project with 20/20 hindsight". One of the big lessons we've learned as an industry involved in software development is to start small, iterate often, get feedback from users. What I see time and again with these types of overblown space elevator projects is a kind of fundamental...immaturity -- the people who run these don't seem to understand how to achieve results with the minimum required to do it.
They've never started small and iterated to large. The projects they've worked on have all been so enormous, and have taken so long, that they have very few data points to draw lessons from. Each iteration and growth cycle in a mature project creates lots of information that can be reapplied elsewhere. But for people who've only ever grown up in large enterprise software projects, that's the only approach they know, and there's only a handful of lessons they've learned.
It's not only architecture astronauts, but the entire ecosystem that supports "enterprise" software engineering: vendors, consultants, scaled agile experts, design tools -- even university programs that churn out enterprise ready engineers. The appearance of these things, and how widespread they are in certain circles, demonstrates how those circles have regressed and tossed into the rubbish bin the lessons that we've learned. The penalty that's being paid is that these same lessons have to be continuously relearned again and again, but instead of pushing against outside models of how to do things (e.g. trying to adopt physical engineering approaches to software), we have to push against our own industry.
When you go against billion dollar enterprise software industries, you end up sounding like a heretic. It's really only when you get a long portfolio of success stories can you succeed. But that's very hard to get in today's climate where people spend their entire careers trying to kill ants with nuclear weapons dropped from the orbit of Mars.
I just didn't put the blame on the previous team, maybe I should have for sticking with it for two entire iterations, still not entirely convinced. I think they couldn't help it and the decisions were already made for them by large faceless corporations. I guess what's the point of calling themselves engineers if they don't feel like they are in a cockpit pressing lots of buttons right from the start?
By profession, I call myself a computer programmer, not an engineer or architect in order to remind myself that I should weigh problems by starting from fundamentals such as a state machine, an integer constraint system or a breadth first search, before setting up service discovery or training neural networks.
I stopped the work, broke the team up (and let a few of them go), re-assigned the goals of the project to 3 people who, working half-time, from scratch, and in Python, rebuilt the entire thing in 3 months, delivered it and it worked (and continues to work). "Builds" are now an scp to a VM, and a restart of a flask application.
My wife recalls a similar experience where a complex application she worked on (and ran reliably on very little infrastructure and with an entire team of 4), and serviced thousands of simultaneous users, was replaced (when a new CTO came in) by a giant, consultant driven Java project that required approximately 6x the infrastructure, 3x the staff to keep it going, and at launch could only serve 9 to 12 users at once. It was a debacle. But sunk cost fallacy forced her company to throw millions of dollars of consultants at it until it achieved some minimal level of barely available service. She and her boss left around that time.
She heard that a couple years later the CTO was let go, the consultants fired and it was all replaced by a team of 5 rewriting it all again from scratch (in Java) but like you also without all the architecture barf dropped in from orbit, and immediately went back to servicing the thousands of users again.
I'm convinced that this is the fault of some kind of consultant industry that seems to infest enterprise software circles that is really only good at inventing ways of designing systems that require more of their services, but never seem to actually deliver working systems.