News for hackers. Stuff that matters.
Re:How to Build Software Like an SRE by jasonhansel on Monday October 17, 2022 @09:24AM (#33233127)
The answer is to "kick retries up the stack"; when you fail to reach a service, you return a 503 and have your clients retry, to avoid a case where every service in the stack starts retrying all at once and causes a massive increase in traffic.

IMHO you should only add retries if it proves necessary in practice to reach your SLA, if you're building something that isn't itself triggered by an RPC, or if you're performing an operation that can never be made idempotent.

Re:How to Build Software Like an SRE by dmux on Monday October 17, 2022 @10:21AM (#33233856)
I like the idea of pushing the retry up to the caller, but in a lot of apps I've seen they've built up an in-memory object-graph from the results of calling out to other services. Wouldn't failing due to one bad service and asking the caller to retry result in every other service being unnecessarily hit?
Re:How to Build Software Like an SRE by CBLT on Monday October 17, 2022 @12:18PM (#33235737)
I had a tech lead who vehemently agreed with the parent commenter (retry from the top), but I ended up learning different lessons.

* Differentiate retryable and non-retryable errors. If the service can't return success because the DB it queries is borked, it should send a non-retryable error. Then it won't get overwhelmed by retries from upstream.

* Retry configuration should have sane defaults. Even "retry once" is too often; many services aren't overprovisioned for 100% increase in traffic. What ended up working here for us was having the retry module collect req/sec statistics, then only allowing 20% of that number in retries/second. Individual requests can be retried twice. That was small enough to not push over any services, but enough retries to compensate for garden-variety unavailability.

* Services shouldn't serve requests first-come-first-served under load. When SHTF, fairness means everyone has to suffer long delays, often much longer than rpc timeout. Instead, serve them in an unfair order - the most recent requests to come in are the most likely to have a caller still interested in them. Answer those!

* Use headless services in kubernetes. By exposing the replicaset to the client, the client can load balance itself more intelligently. Retries should go to different replicas than the failing request. Furthermore, you can perform request hedging to different replicas than the lagging request.

* Define a degraded form of your in-memory object-graph. If a feature is optional to the core business flow, it shouldn't take down your whole product. This one is a lot more involved. We needed custom monitoring for degraded responses, in-memory collection and storage of "guesses" to substitute for degraded portions of the object graph, as well as some other work I can't think of right now. This does enable an organization to compartmentalize better, having faster, less fearful deploys of newer initiatives.

Re:How to Build Software Like an SRE by sokoloff on Monday October 17, 2022 @03:07PM (#33238148)
It seems like knowing the difference between a retryable and non-retryable error is itself difficult (perhaps impossible in practice).

DB is unreachable? Is it the DB or my host? If it’s the DB, maybe it’s not retryable, but if it’s me (and I’m load-balanced), it is probably retryable.

Re:How to Build Software Like an SRE by savy91 on Wednesday October 19, 2022 @05:59AM (#33259834)
IMHO even if db is not reachable, it should be retriable.
Re:How to Build Software Like an SRE by electroly on Monday October 17, 2022 @10:34AM (#33234054)
Sometimes you want the exact opposite, though. Consider an endpoint that makes 100 behind-the-scenes requests (say, to S3). You absolutely want to retry at the lowest level, not the highest level. You could fail on the 99th request. If you kick it up, the caller will retry and you'll do those 99 requests again, instead of just retrying the one that failed. With enough requests, there's a point where you're unlikely ever to succeed without one of the calls failing if you restart from the beginning every time. I don't think you can "one size fits all" this, and that's one of the reasons retries are hard.

I use S3 as an example because it has a comparatively high random failure rate. You MUST implement timeouts and retries if you're using S3 as a backend.

Re:How to Build Software Like an SRE by jasonhansel on Monday October 17, 2022 @12:12PM (#33235643)
That's why I said "except if your SLA requires it." If you've agreed--with other teams, with outside clients, or just as a project goal--that your service should work on 99.9% of calls, and you find in practice that you need to retry S3 calls to meet that target, then adding retries is reasonable.

If the problem is just that the 99 calls could overload downstream services on retries, the ideal solution is to add rate-limiting, though admittedly that is an imprecise science.

Re:How to Build Software Like an SRE by benlivengood on Monday October 17, 2022 @06:41PM (#33240438)
There's never a reason that a human needs to read an error message caused merely by Kubernetes restarting a pod in the normal course of operations.

Passing every error up the stack causes people to be confused by 500 status errors for trivially retry-able calls. Browsers (and TCP) natively implement retries despite there being no SLAs in place.

Re:How to Build Software Like an SRE by zffr on Monday October 17, 2022 @01:38PM (#33237057)
If a client retries it’s not necessary for all 100 requests to be attempted again. The system could be designed to only repeat the unsuccessful operations.
Re:How to Build Software Like an SRE by JustLurking2022 on Monday October 17, 2022 @02:44PM (#33237893)
You mean like retrying at the lowest level?
Re:How to Build Software Like an SRE by vlovich123 on Monday October 17, 2022 @08:46PM (#33241609)
You do want the policy on retries to live and be evaluated at the outermost level where your business logic lives. If that’s living across an RPC boundary, then you’re stuck making this weird trade off and where I think this back and forth is happening - people have different mental models of the specific service they have familiarity with that they’re using to test the recommendation against and because there’s a trade off to be made, the advice can be correct in some contexts and wrong in others. If you can encode your policy and it’s evaluation generically so it flows through the stack, that’s not terrible although it becomes hard to manage certain other kinds of SLAs (eg bounding latency of your overall operation or the latency of a specific suboperation).
Re:How to Build Software Like an SRE by geocar on Monday October 17, 2022 @03:20PM (#33238296)
> Consider an endpoint that makes 100 behind-the-scenes requests

Don't do that; Don't ever make middleware retry.

An API endpoint should be quick, and if you can't figure out how to make it quick, use batch processing metaphors (submit task/query task) and a task-runner instead. The task-runner can retry indefinitely, but a user needs to be given feedback or they will push the refresh button.

Re:How to Build Software Like an SRE by qu4z-2 on Monday October 17, 2022 @06:13PM (#33240189)
Lisp's conditions and restarts are great for this, so you can make the policy decision at a high level while not fully unrolling the stack and allowing retries to be done in the original context if that's the verdict.
Re:How to Build Software Like an SRE by mcguire on Monday October 17, 2022 @03:19PM (#33238287)
That's a variation of ye olde end-to-end argument.

In fact, this subthread seems to be recreating many of the network protocol design ideas from the ancient days.