Here's a quick intro to Elixir (it takes about an hour), where you'll build a distributed app and have it automatically recover from a failure in a process: http://howistart.org/posts/elixir/1
* I had 400,000 processes running on a modest VM without maxing out.
This is a good pattern (one of many), but it is no silver bullet, either. If you overdo this, you end up with overly defensive programming, where all code is littered with catching theoretical error conditions that will never occur. That hides the actual functionality and makes code really hard to read and to change.
The solution pattern for this is to move checks at the boundary of the module, so that the code inside can make safe assumptions on the data it receives, so it can concentrate on the real functionality.
As the article says, whatever you do, you can overdo. Software architecture is always about balancing and judging conflicting appoaches and goals.
I think functional languages have a more healthy appreciation for layering by nature of requiring functions to act on data rather than having mutable data types. I believe if we want to reduce complexity we HAVE to create more layered systems to scope the bounds of our complexity. The problem is many people see this as over-architecting things. With time, a design can evolve such that 10 or 20 layers, as compared to the 3 in MVC we often see, could be manageable with help from our languages and environments (IDE / text editor / etc.).
Looking at biological systems though, that is the opposite of how reliable systems are constructed. If we design from the assumption of failure and runaway complexity, with watchdog and recovery processes, and using emergent behavior from small discrete components to obtain large-scale results, then we can build software components that are much better able to recover from unexpected interaction and produce more reliable systems. It seems to me that this is what the theory should be. I just have no idea how to do that in the real world when building a web app, aside from recycling server processes every now and then.