Re:Gremlin Free – Run chaos experiments to prevent outages
by flowardnut on Wednesday February 27, 2019 @05:30PM
(#19267530)
idempotent requests, stateless services, etc are all parts of a fault tolerant system.
your service has a few ways to deal with a dependency going down -- maybe it's a retry, maybe it's opening a circuit breaker and returning a default payload instead of calling that service.
It really depends on what specifically the service is and what it's calling (so it's a very case by case issue).
One of the very neat features of istio is that you can do this tuning in real time -- spin up your services, simulate faults, and then test your service while tuning your retry logic to see what the best user experience is.
your service has a few ways to deal with a dependency going down -- maybe it's a retry, maybe it's opening a circuit breaker and returning a default payload instead of calling that service.
It really depends on what specifically the service is and what it's calling (so it's a very case by case issue).
One of the very neat features of istio is that you can do this tuning in real time -- spin up your services, simulate faults, and then test your service while tuning your retry logic to see what the best user experience is.