Disk: essentially limitless. If disk of VM runs out, node will crash, new node will start. Service should keep running on other nodes meanwhile. If restarts happen too often, throughput/error rate will suffer or cost will rise.
Modern mentality is "let it crash" [0]. Software has bugs. Design so it can crash and scale depending on need.
[0] https://medium.com/@vamsimokari/erlang-let-it-crash-philosop...
And "let it crash" is hardly "My service is invincible!" I'm sure any serious Erlang deployment has seen what I've seen as well, which is a sufficiently buggy process that crashes often enough that the system never gets into a stable state of handling requests. "Let it crash" is hardly license to write shitty code and let the supervisor pick up the pieces. It's a great backstop, it's a shitty foundation.
I haven't used Erlang, but my impression is that it's probably the same experience there?
In my case, IIRC, it was something to the effect of, a lot of our old clients out in the field connected with a field that crashed the handler. Enough of these were connecting that the supervisor was flipping out and restarting things (even working things) because it thought there was a systemic issue. (I mean, it was right, though I had it configured to be a bit to aggressive in its response.) The fact that I could let things crash didn't rescue the fact my system, even if I fixed that config, would strain its resources constantly doing TLS negotiations and then immediately crashing out what were supposed to be long term connections.
Obviously, the problem was in the test suite, we were able to roll back, address the problem, ultimately this was a blip not a catastrophe. I just cite it as a case where "let it crash" didn't save me, because the thing that was crashing was just too broken. You still have to write good code in Erlang. It may survive not having "great" code but it's not a license to not care.
Many software development processes considered as a whole are anti-fragile... mine certainly is, which is a great deal of why I love automated testing so much (I often phrase it as "it gives me monotonic forward progress" but "it gives me antifragility" is a reasonably close spin on it too)... but that's not unique to Erlang nor does Erlang have anything special to help with it as everything Erlang has for robustness is focused at run time. You can have anti-fragile development processes with any language. (The fact that I successfully left Erlang for another language entirely is also testament to that. I had to replace Erlang's robustness but I didn't have to replace its anti-fragility, since it didn't have it particularly.)
It's not the tool, it's how you use it.
However there are industries applying these lessons in HPC / data analytics / things that touch money live .. operating on scales of users in the 10s to maybe 100s. So stuff where downtime is far more costly both in dollars and reputation.
I'm also intrigued by the constant cloud refrain of "stuff crashes all the time so just expect it to" coming from a background where I have apps that run without crash for 6 months at a time, or essentially until the next release.
I'm all for scaling, recovery, etc.. I just fail to understand why it is desirable for this to be an OR rather than an AND.
What if stuff was highly recoverable and scalable but also.. we just didn't run out of disk needlessly?
IMHO, those aren't mutually exclusive. Your app code should be robust enough to run 6+ months at at time, and the "stuff crashes all the time so just expect it to" attitude should be reserved for stuff outside your control, like hardware failures.
It pertains to e.g. single database connection, single HTTP request etc. If something crashes there your APM reports it and Erlang's unique runtime continues unfazed. It's a property of the BEAM VM that no other runtime possesses.
"Let it crash" is in fact "break your app's runtime logic to many small pieces each of which is independent and an error in any single one does not impact the others".
Scaling an Erlang node is very rarely the solution unless you literally run out of system resources.
RE: error handling, Elixir apps I coded and maintained never had a problem. Everything was 100% transparent which is another huge plus.
In general CGI and PHP had the right idea from the start but the actual implementations were (maybe still are? no idea) subpar.
Erlang's runtime is of course nowhere near what you will get with Rust and very careful programming with the tokio async runtime, but it's the best out there in the land of the dynamic languages. I've tried 10+ languages which is of course not technically exhaustive but I was finally satisfied with making super parallel _and_ resilient workloads when I tried Elixir.
For a lot of tasks I can just code in the Elixir REPL and crank out a solution in an hour, including copy-pasting from the REPL and into an actual program with tests. Same task in Rust took me a week, just saying (though I am still not a senior in Rust; that's absolutely a factor as well) and about 3/4 of a day in Golang.
The only other small-friction no-BS language and ecosystem that came kinda close for me is Golang. I like that one a lot but you have to be very careful when you do true parallelism or you have to reach for a good number of libraries to avoid several sadly very common footguns.
Please read the last three paragraphs in [0]: "a well-designed application is a hierarchy of supervisor and worker processes" and "It handles what makes sense, and allows itself to terminate with an error ("crash") in other cases."
I've personally designed and co-implemented mission critical real-time logistics systems which dealt with tens of thousands of events per second, with hundreds of different microservices deployed on a cluster of 14 heavy nodes. Highly complex logic. At first we were baby-sitting the system, until it became resilient by itself. Stuff crashed all the time. Functional logic was still intact. Then we had true silence for months on our pager alerts.
I call it anti-fragile and Taleb is right. You can't make a system resilient if you don't allow it to fail.
However theres lots of cloud people who don't know linux.
Therein lies the challenge. Too many of these cloud devs forget that whether its "server less" or not, theres a .. server, somewhere.
All the layers of abstraction work when they work, and when they don't .. best of luck figuring out what's going on in a timely fashion. Hope you aren't running any money on top of it.
Yeah observability is great, but no one I've seen in cloud manages to launch with it. So in place of SRE org with full observability stack, they launch with effectively nothing.
Strong attitudes of "we don't measure hardware metrics like cpu/memory/disk, we measure customer facing stuff like throughput/error rates etc". Sounds good, but what do you think happens to those if you run out of disk bro? Pretty sure the running out of disk happens first. Imagine if you could like.. catch it?