Server crashes happen. This forces you to deal with them instead of pretending they won’t.

Well yes, but I would suggest that they are uncommon enough that a few requests failing isn't a problem when those happen.

It's an entirely different story when you are killing processes constantly.

If "a few requests failing isn't a problem" is a reasonable statement in your company then this type of service isn't really aimed at you. The point is to help shake out fault tolerance issues. If you can already tolerate faults there isn't much of an issue.

It doesn’t _need_ to be constantly. The important part is that it is done _deliberately_ to understand what happens when failures occur.

idempotent requests, stateless services, etc are all parts of a fault tolerant system.

your service has a few ways to deal with a dependency going down -- maybe it's a retry, maybe it's opening a circuit breaker and returning a default payload instead of calling that service.

It really depends on what specifically the service is and what it's calling (so it's a very case by case issue).

One of the very neat features of istio is that you can do this tuning in real time -- spin up your services, simulate faults, and then test your service while tuning your retry logic to see what the best user experience is.

well for example in our systems all api calls only moves from a know state to another known state and any call failure redirects the client/user to the dashboard trough an error handler so they have to reload the last good state saved on the database.

not perfect, but having a server crash is not much different than having a connection reset by a wifi status change or an upload timing out due the mobile network going away or the user navigating away or closing the browser.

It sounds like you are saying "The in-flight requests fail" to me.

I really don't like the idea of saying that it's simply okay to give random users a bad user experience like that when you are actually killing servers yourself all the time.

It's a different approach to managing risk -- minimizing impact of failure rather than minimizing the likelihood of failure.

It's nice to know that you can kill a process and the only impact is that in-flight requests fail, rather than having a more significant outage if a process crashes and the failover doesn't work, or the process doesn't automatically restart, etc.

If you accept that requests will fail you can build retries into the system. It's a lot harder to make a system more resilient if you avoid testing the failure scenarios.

Exactly! Chaos engineering is all about thoughtfully planned out experiments, to observe what the user experience will be when something fails. Doing this on your own terms allows you to improve the experience so that your customers aren't affected.

You can decide what happens when an in-flight request is dropped, whether you hold onto the state somehow and retry or the client could fail gracefully with a relevant error message.

Another thing that's not often caught by "normal" testing but that chaos engineering can capture is when multiple things fail together in random ways. It can be surprising how otherwise robust services can fail badly when multiple things go wrong at once.

When you have nested service calls, a single downstream failure shouldn’t fail all the way back to the root request, in most cases.