News for hackers. Stuff that matters.
Re:Using load shedding to avoid overload by tyingq on Sunday October 10, 2021 @11:04AM (#28818963)
The article does mention prioritization, but doesn't mention my favorite pattern with this. A priority queue that favors end users "farther down the process" is nice for load shedding.

Like, for an ecommerce site, being able to prioritize users in the checkout path first, not-in-checkout but non-empty cart second, etc.

Or prioritizing features in a similar way. Turning off, for example, "people that bought this, bought that", under load.

Re:Using load shedding to avoid overload by dmurray on Sunday October 10, 2021 @02:41PM (#28820893)
> doesn't mention my favorite pattern with this. A priority queue that favors end users "farther down the process" is nice for load shedding.

The (fantastic) article suggests precisely this:

> let's say a service has two APIs: start() and end(). In order to finish their work, clients need to be able to call both APIs. In this case, the service should prioritize end() requests over start() requests

Re:Using load shedding to avoid overload by dyanacek on Sunday October 10, 2021 @12:02PM (#28819455)
This is a great way to describe it! I gave a similar example of pagination and how the later pages might be better to prioritize over initial pagination requests, but your example is a nicer illustration. Thanks for that!

There’s also someone I was talking to after writing the article who said they can fall back to statically rendered versions of certain pages on Amazon.com during overload. The trick is to have a page that is still useful!

And for the “turning off features” idea - this happens today on Amazon.com. If a feature on the site fails to render successfully or on time, it’s left off of the page. Critical functionality can be left off, so it’s a judgement call on what’s allowed to fail the page render.

Re:Using load shedding to avoid overload by tyingq on Sunday October 10, 2021 @12:34PM (#28819718)
Ah, yes, you're right...I missed the pagination example fitting that pattern.

"If a feature on the site fails to render successfully or on time, it’s left off of the page. Critical functionality can be left off, so it’s a judgement call on what’s allowed to fail the page render."

Oh, that's useful also, but I meant a step farther where the page doesn't ask for those widgets if (load > X). Which avoids calling it at all.

Re:Using load shedding to avoid overload by dyanacek on Sunday October 10, 2021 @01:00PM (#28819942)
Good point around avoiding the call in the first place. This is a very tricky topic, I’ve found. Things that try to guess the nuanced health of a dependency can lead to outages when they guess wrong. These circuit breakers are helpful if they’re right, but harmful if they’re wrong.

For example, say a service is backed by a partitioned cache cluster, where the data is hashed to a particular cache node. Now let’s say one node has a problem, causing requests to data that lives on that node to fail, but others to succeed. If a client is making requests for data that happens to live on all nodes (the client doesn’t know about these nodes by the way, it’s just an implementation detail of the service) and sees an increased error rate, should it start failing some requests? It could take a single partition outage and increase the scope of impact into a full outage.

Anyway I’ve been meaning to write an Amazon Builders’ Library article on this topic, or to convince someone else to do it (looking at you, Marc Brooker!)

Re:Using load shedding to avoid overload by vinay_ys on Sunday October 10, 2021 @01:26PM (#28820202)
Graceful degradation by turning off optional features and making the critical path (order completion funnel) as lightweight as possible by reducing costs (# of results fetched/scored, turning off reco etc) was the best way to scale systems seeing 3x higher peaks year over year; and 10x higher peak during flash sales compared to normal periods, especially without breaking the bank trying to provision that much capacity.

p.s. - btw, no cloud has that much elastic capacity.

Re:Using load shedding to avoid overload by zodvik on Sunday October 10, 2021 @02:18PM (#28820699)
I saw this comment and thought this exactly described what my $previous_company did. Then, I saw the username :-)
Re:Using load shedding to avoid overload by kawsper on Sunday October 10, 2021 @04:58PM (#28821923)
> turning off reco

What is reco?

Re:Using load shedding to avoid overload by iampims on Sunday October 10, 2021 @05:36PM (#28822213)
Recommendations.
Re:Using load shedding to avoid overload by ignoramous on Sunday October 10, 2021 @07:18PM (#28822837)
> A priority queue that favors end users "farther down the process" is nice for load shedding.

Another pattern from the networking world, one which Facebook wrote about, is CoDel aka Controlled Delay (switches tasks to lower timeouts when the work-queue begins to build-up) combined with adaptive LIFO (processes last-in tasks first when under load, but FIFO otherwise): https://queue.acm.org/detail.cfm?id=2839461

Re:Using load shedding to avoid overload by noisy_boy on Sunday October 10, 2021 @10:24PM (#28823987)
This makes me think about APIs that face a variety of queries (probably not e-commerce where patterns are more predictable but think internal warehouse) - some queries may be more "expensive" than others (e.g. they query data that touches a big percentage of the underlying data based on criteria where database index is not feasible). What if instead of allowing all requests to go to the API directly, we let them pass through a grader, "good" queries (say ones that are using indexed columns) get a higher "grade" and then we put the requests in a priority queue sorted by grades and descending time (so if grades are same, newer queries get processed first). That becomes the work queue. Interested to hear what people think about it / if someone is using something similar and their experience with it.
Re:Using load shedding to avoid overload by withinboredom on Monday October 11, 2021 @12:20PM (#28829988)
TBH, the specific approach sounds like a bad one. You can easily swamp a server with small requests and the big ones will never run, which are probably important. It would just incentivize engineers to write smaller queries while your older (Unmaintained ones) ones would never run.

Same with deprioritizing queries that are older in the queue. You’d end up deprioritizing people who’ve already been waiting, so they’d likely try again instead of waiting.

Thus you end up with the feedback loop this article talks about, where you’re amplifying the problem instead of addressing it.

Re:Using load shedding to avoid overload by lokar on Sunday October 10, 2021 @12:36PM (#28819738)
The generic goal I think you are describing is to maximize goodput.

Another simple approach is to serve queues (of like requests) in LIFO order.

Re:Using load shedding to avoid overload by dyanacek on Sunday October 10, 2021 @01:04PM (#28819978)
Yes! LIFO is a fantastic improvement. This suggestion is buried in the article a bit. Maybe I should have elevated it, or maybe broken it into more pieces. There’s so much to talk about on this topic. But yeah, LIFO is totally “This one weird trick that will make your service bulletproof to overload! Chaos Monkeys hate it!”
Re:Using load shedding to avoid overload by tome on Sunday October 10, 2021 @02:14PM (#28820671)
I'm really confused about how and why this works. Why is it a good idea to keep really old requests unhandled? I feel like I must be missing something obvious.
Re:Using load shedding to avoid overload by 8note on Sunday October 10, 2021 @05:42PM (#28822247)
A queue distributes the latency increase to all requests whereas a stack only increases the latency for some requests when you're overloaded.

This means that if you catch up to the incoming new requests, a queue keeps every request running slower due to the time spent in the queue.

The steady state stack on the other hand, gives the same couple items worse and worse latency, while all the new items go back to normal.

Chances are the long latency requests will be retried, and there's a roughly fixed number of items to be retried, so you don't have to worry too much about the ones stuck in the stack. For a queue, the retries lengthen the queue, and that waiting time is added to every item, making them more likely to retry too, lengthening the queue further etc

Having the stack leak items at the bottom gets the same benefit - you don't have to fulfil them, but if you can get the items back out of the stack quickly, it's still worth working on and completing them before the client needs to send a retry. The more of them you get in, the more 9s you get to look more like your p50 than your p100

Re:Using load shedding to avoid overload by lokar on Sunday October 10, 2021 @11:49PM (#28824448)
Consider the two steady states:

You are keeping up and the queue is mostly empty, the order does not matter

You are not keeping up, the queue is growing. If nothing changes, the age of items removed from the front will grow and grow, and eventually all be timeouts or abandoned.

Re:Using load shedding to avoid overload by tome on Monday October 11, 2021 @08:56AM (#28827790)
Oh, I think I see. With a queue the failure state is that everything times out. With a stack you risk sacrificing only the oldest requests and keep the newest alive.
Re:Using load shedding to avoid overload by lokar on Monday October 11, 2021 @10:31AM (#28828734)
And you end up not servicing some random sample that get buried in the stack before you can get to them.
Re:Using load shedding to avoid overload by oriolid on Sunday October 10, 2021 @03:25PM (#28821182)
It guarantees that in overload situation the requests that get handled are handled quickly. In the same situation a FIFO would grow until all requests are really slow or time out without increasing throughput. The reason to keep old requests in LIFO instead of dropping then right away is that they can be served when load drops, just in case there's still someone waiting for the page to load.
Re:Using load shedding to avoid overload by tyingq on Sunday October 10, 2021 @02:23PM (#28820743)
I suppose the older requests are less likely to have actual people doing a page reload/retry that stacks up yet more demand.