There are lots of examples of successful companies using microservices, but I believe the real problem is in defining what constitutes a microservice. Most people call things "microservices" that are nothing of the sort. I can unequivocally say if you built a "service" that depends on other things being 100% available (like another "service") than you haven't built a microservice (ie: those things you built shouldn't be called services).
By that token, autonomy is a pretty important factor. The Udi Dahan teachings (https://particular.net/adsd) (currently available for free) promote this style of architecture. A concrete example of a toolkit for building true microservices can be found in Message DB (https://github.com/message-db/message-db) and/or Eventide (http://docs.eventide-project.org/)
I wouldn't suggest, however, that anyone can just watch the course, pick up these tools and succeed. Like baking a good loaf of bread, it takes a lot of skill, work and experience. Whether or not you succeed at building microservices is ultimately up to you and your team.
Netflix runs quite well in practice. I think they do redundant service calls, what is the only minimally sane way to develop a distributed system. The funny thing is that I have never seen any serious discussion of redundant calls, except for it being implicit on practical designs on the anecdotal "how it works for us" articles that pop once in a while. Most times people won't even discuss redundant servers. I imagine everybody thinks it's obvious, and well, I would agree, except for the fact that most people I see do not think so.
But well, Netflix couldn't avoid having a distributed system, so they aren't really representative for nearly anybody.
But the reason I say this is about the team is because I've seen a well built, well groomed service be passed off to an outside team and immediately turned into a disaster. Same service, same business case, just a team without the DevOps savvy and willingness to follow the patterns.
Netflix does some.. unusual things with microservices, mostly with how they treat version rollovers. It's not bad or good, but it looks different from how many other shops handle the same problem and it means asking the question "is everything working?" is extremely difficult but asking the question "how many people are able to start watching?" is pretty easy.
That doesn't seem true. I would imagine that at Netflix scale, you probably have request tracing libraries that can give you a graph of service dependencies. Whether it's worthwhile to consume that, or easier to just let Chaos Monkey run rampant is another question.
Also, I very rarely have issues with Netflix. Typically when I do I can just exit the stream and restart it. Anecdata, but I could count on one hand the number of times I've had a title just not play, or Netflix be down entirely.
They don’t measure how often 100% or their services are up because perfect uptime is not the goal and is too expensive (if it’s even possible). If an internally facing service being down doesn’t affect a core metric like number of stream starts by customers, then it’s foolish to treat it as needing 99.999% uptime.
You answered your own question with Netflix. While you're right that it's not clear Netflix would've needed to develop their chaos monkey tooling and the like, it's not clear at all at that an equivalent system is possible as a monolith. Even if a monolithic Netflix system were technically possible, it's not clear if a monolithic system would be organizationally feasible either (Conway's Law)
My point being that if and only if we have a track record as an entire field of making decisions based on case studies, AND if case studies have a track record of being objective rather than proffered as a result of a marketing agenda, then the question is ultimately legitimate.
There are more shops by total count that fail with monolithic architecture. That's inevitable just based on the infinitesimal number of projects executed as service architectures rather than monoliths at large in the wild. But still, we carry on with monolithic architectural style as if its outcomes were assured.
It's far easier for the vast majority of developers to build a monolith because it allows development to proceed without having to have any knowledge of or practice with the tricks and traps of distributed systems - an entire body of knowledge that a developer might never get meaningful and practical exposure to for an entire career.
The trouble starts when microservices are attempted by developers who can't imagine that there are entire bodies of software development knowledge that developers aren't presently in possession of.
Microservices is just, as Adrian Cockroft used to say, "Service-Oriented Architecture with bounded contexts".
Web development is absolutely not a preparatory course in service oriented architecture. But the vast majority of web developers who attempt to take on SOA while simultaneously presuming an omniscience in all things software development due to their experiences only with monolithic web development will often fail to build a SOA. They usually end up with something that isn't quite SOA and isn't quite a monolith. And that's where the failures largely come from.
I work exclusively in microservices and SOA, and have since 2015. Before that, I worked principally as a web app developer, and did some work off-and-on in SOA implementations. And before that I spent years becoming oriented to the architecture. I don't make the mistakes that web developers typically do when they presume that web development knowledge is a sufficient prerequisite for working in SOA.
So, it's not a question of whether an architectural style works or doesn't. The majority of failures in microservices can be attributed to ignorance and to the narcissism that is permissive of it.
So, I would ask this question instead: Are there any cases where developer over-confidence and over-simplification went well?
These qualities don't tend to serve any architectural style well.
I've never heard of a well-designed SOA not going well. Every single case of microservice project remediation that I've participated in had as the most significant contributing factor an utter disregard for the body of knowledge that the microservices architectural style is built upon.
There are a lot of things that developers can get away with when doing the kinds of tinkering and wandering that typifies typical web development work. But those things don't work once we cross the line into SOA. And unfortunately, the incessant chasing after trivial resumé candy hasn't prepared the average developer for the rigorous mindset needed for SOA work.
As "microservices" became to next fad for perennial fad chasers of the software development world, they finally encountered a kind of work that they could not get away with by faking it. And so, we see a lot of failures. But the vast majority of the failures are personal failures and character failures, rather than failures of an architectural style.
The fat part of the developer bell curve was simply overreaching when it presumed to try to get away with building service architectures with the same level of disinterest in architecture and process that we can get away with in typical web development. Like a kid with copious experience building kites presuming to strap themselves to a hang glider and just "going for it". The outcomes are mostly predictable.
In the end, if a little time is invested in learning the fundamentals that have so far been eschewed for the sake of expediency, anyone can succeed with SOA and microservices. It's not that the realities of the architectural style are unlearnable, but it can't be arrived at by the level of tinkering and wandering that we can just get away with in monolithic web development.
In every service architecture I've worked with in the last few years, you could theoretically run every single "service" within the same physical process -- provided they all shared the same runtime/lang the way a monolith does. I tend to start service architectures using Ruby exclusively with the Eventide toolkit, so this is actually a viable approach for most teams I've worked with. But it's never ultimately made sense. Weighing the pros/cons, a consolidated deployment topology wouldn't add any benefit, and it would actually make it far more difficult for the operations folks, practically speaking.
I've helped put services into production that 1. carry out crucial, "core" business logic reliably and efficiently, 2. can be scaled horizontally without changing the code, 3. never raise exceptions or cause outages, and 4. don't need to be touched for years because new features are more naturally composed around them anyways (i.e. open/closed principle). Practically speaking, it's quite a bit easier for human programmers to reach this degree of precision with a smaller program than with a much larger one. And if you add up a lot of these high quality programs, you get a high quality system. Building a high quality system out of a single program is much, much harder for humans in practice.
I'll use crude terms here for a second: every good service put into production brings a net benefit to the organization relative to the same code having been entangled with an existing pile of code. That may sound like a "No True Scotsman" fallacy, but the definition of "good" in this context is precisely the net benefit being added. If you build and deploy services that have a high degree of quality, you get a corresponding benefit. If you build and deploy programs that _don't_ stand on their own, _don't_ leverage durable messaging, and take other programs down with them when they crash, then you get a rather large mess on your hands. In fact, you never had a service architecture at all; we call this failure mode a "distributed monolith."
I'll acknowledge that most attempts at microservice architectures in the wild don't seem to succeed. Anecdotally, they are particularly prone to failure when their architects don't understand the underlying principles of distributed systems all that well; they neglect important considerations like messaging idempotence, the deleterious effect of synchronous request/response messaging, and the need for deliberate, thoughtful design. In other words, they build systems out of N number of microservices, and get N^2 fault tolerance scenarios, which you adeptly called out for being foolish. They're arguably worse off than if they went with a monolith, but neither would be an architecture I'd personally want to work with.
Though the transition is still in progress, I can say that the path forward is clear and hopeful. This company had previously explored other "SOA" paths (distributed monolith) and it was clear that those were very problematic. Luckily, we were able to steer them to an actual evented architecture.
As I said, it's still early days for this particular project, but if you're feeling your monolith is getting harder and harder to develop on, you should start looking into evented architectures. Eventide (ruby)/messagedb (postgresql) are awesome technologies and would be the first I'd consider. Eventide also has a nice slack community filled with people that are learning and improving their software design skill both in the large (architecture) and the small (individual classes/etc). It's small, but they're good people.
Not to say that wouldn't happen for SPOF microservices (e.g. your auth servers), but the surface area is potentially larger for monoliths.
From an end user perspective, Netflix runs in “constantly degraded” mod.
From an engineering perspective, they track “number of successful stream starts”, instead of percentage of the time 100% of their services are working. That’s a huge red flag.
As a researcher, the monitoring and fault-propagation / modeling work they’ve done to get it to stay up at all is impressive, but it’s not clear all of that tooling would be necessary if they didn’t have to reason about N^2 fault tolerance scenarios, where N = 100’s of microservices. That’s on the order of one fault tolerance scenario for each atom in the universe.