How Discord handles over a million requests per minute with Elixir’s GenStage

Re:How Discord handles over a million requests per minute wi... by teej on Monday December 12, 2016 @05:50PM (#13162794)

That seems too important to have zero visibility on to me. Just eyeing the graphs, your queue size grew at 750m/s from 17:49 to 17:50. You then starting shedding at 17:50 for 40s. Assuming the ingress rate was roughly linear (which it looks like it was) you shed ~30,000 requests out of 3-4M. Does that not seem high to you?

This system seems great for at most once delivery. I wish I had more problems to solve with that constraint.

[ Parent | on HN ]

Re:How Discord handles over a million requests per minute wi... by Sikul on Monday December 12, 2016 @07:34PM (#13163465)

Yep, such is life with new tech. GenStage was 0.3.0 when we started using it. We'll be able to get this visibility once we update to a newer version (hasn't been prioritized).

FWIW, the buffer only fills up to the peak about once a month. Load shedding is the last ditch effort in catastrophic situations so that everything doesn't fall apart.

[ Parent | on HN ]

Re:How Discord handles over a million requests per minute wi... by henrygrew on Tuesday December 13, 2016 @08:25AM (#13166569)

I don't find load shedding as acceptable, perhaps if there was a criteria for dropping based on priority it would make more sense.

[ Parent | on HN ]

Re:How Discord handles over a million requests per minute wi... by Sikul on Tuesday December 13, 2016 @02:58PM (#13170336)

I think it depends on your problem space.

[ Parent | on HN ]