News for hackers. Stuff that matters.
Re:Preventing server overload: limit requests being processed by Animats on Monday August 07, 2017 @06:24PM (#14951770)
I've used fair queuing to solve this problem.

I have a site and API which rates other sites, checking them for ad links and trying to match them to real world business records. Inspecting a site takes 10 seconds to 2 minutes. The API, which is usually used from browser add-ons which tag web search results, lets anyone request the rating info for a site.

Most requests are already cached and return immediately. Requests for unknown sites are queued up for the rating engine, which runs in processes separate from the web-facing side, and return "wait" to the requestor. So there's a work queue to manage.

It's managed using fair queuing. If there is no pending request from the requesting IP address, the new request goes into the queue. If there's a request from the same IP address already in the queue, the new request is held until the first one completes, and then added to the end of the main queue. No request is rejected until there are 100 requests from a single IP address. So each IP address competes against itself, and no one IP can hog the system.

A typical overload comes when someone searches for an unusual topic and flips through many pages of search results quickly. This can result in the rating engine having fifty or so new sites to examine within a few seconds. Those requests will be processed one at a time until the rating engine catches up.

This queuing system held up well when someone tried a test where they fed a huge list of sites into the API without waiting for completion of any of their requests. The rating engine ran busily for a week, but they were only tying up one rating process, and it didn't affect other users at all.

There are no adjustment parameters. It just runs. It's not perfect, but it deals well with legitimate transients and with abuse from a small number of IP addresses.

Re:Preventing server overload: limit requests being processed by evanj on Tuesday August 08, 2017 @09:19AM (#14957499)
Fair queuing is an interesting and complementary approach that I have not considered, thanks for the suggestion.

I usually work with "internal" systems that don't always have an obvious "user" identifier, but it would be fairly easy to add one of some sort (e.g. an application id or similar). However, for that to be useful, there still has to be some sort of limit. In your case, the limit seems to be 1 concurrent request per user, up to a maximum of 100. In the case of some internal system, like say a metrics aggregator, what is the correct limit? I'm not sure, and I think it would require tuning.

That said, given a reasonable limit, this would definitely ensure that the "misbehaving" application (e.g. the one flooding the metrics aggregator with packets) is the one that gets punished, rather than everyone else. I'll have to think about this more. Thanks!

Re:Preventing server overload: limit requests being processed by thunderbong on Monday August 07, 2017 @10:27PM (#14953294)
That's really interesting. Can you tell more about how it is implemented technology wise?
Re:Preventing server overload: limit requests being processed by Animats on Monday August 07, 2017 @10:38PM (#14953407)
Python and MySQL/MariaDB. The queue is an in-memory table. Fair queuing is implemented as a big SQL SELECT statement.

The time scale is slow enough that polling it once a second for new work is sufficient. If it had to go fast, it would require something like a standard fair queuing implementation, where you hash the IP address and put the request onto 1 of N queues, serviced round-robin.

Re:Preventing server overload: limit requests being processed by javajosh on Monday August 07, 2017 @10:46PM (#14953484)
Doesn't matter all that much. For ingestion, you might have one VM running one Java process and one DB process as a microservice (I like Dropwizard and PostgreSQL for this). The worker itself would be another Java process with a 100 thread max thread pool that either periodically checks for new work, or uses the new NOTIFY semantics new to Postgres 9.6. In this case the "secret sauce" is that the worker has some extra smarts to reorder the queue to avoid resource hogs, which is the cool part.
Re:Preventing server overload: limit requests being processed by EGreg on Monday August 07, 2017 @07:38PM (#14952214)
What does the rating consist of, and what is its purpose?