Re:Ask HN: How would you queue and process 10K+ long running...
by sethammons on Tuesday November 27, 2018 @01:09PM
(#18544881)
You have an interesting requirement: each job only gets picked up by 1 worker and run once (with up to an hour long job).
I'll contend that you can't do that. You could get at least once or at most once. Let's say that you go with at least once.
Just use a queue like RabbitMQ. Workers connect, request work, ack that they are done, and you should be good to go. Done. You can set this up today if you want.
If you need more thorough duplicate detection, you could sprinkle in some redis to store job state (in progress / complete). Using atomic operations like INCR/DECR on your key, you could pull a job from Rabbit (or your queue of choice), hit redis to ensure that the job is not in progress or already complete due to a network error between Rabbit and your workers, and then proceed appropriately.
The key problem here is that the network could drop requests. You could pull from your queue, complete the work, and think you ack'd, but the queue never gets the ack, so it hands out the work again to a new worker after the lock expires on it. So you could mitigate that with an additional layer of a distributed KV store. But that could have the same problem.
I run a system that processes billions of events a day and we use a system very similar to what I described above (though we have a custom queue solution and a pool of redis nodes that we have some custom quorum logic around). We don't seem to be duplicating hardly any jobs (maybe a handful a week).
They way they do it is by controlling the client and the server and sprinkling some write-ahead-logs with logical clocks and using a formal consensus protocol (paxos) under the hood. Even with all that, I'm skeptical of the exactly once claim.
I'll contend that you can't do that. You could get at least once or at most once. Let's say that you go with at least once.
Just use a queue like RabbitMQ. Workers connect, request work, ack that they are done, and you should be good to go. Done. You can set this up today if you want.
If you need more thorough duplicate detection, you could sprinkle in some redis to store job state (in progress / complete). Using atomic operations like INCR/DECR on your key, you could pull a job from Rabbit (or your queue of choice), hit redis to ensure that the job is not in progress or already complete due to a network error between Rabbit and your workers, and then proceed appropriately.
The key problem here is that the network could drop requests. You could pull from your queue, complete the work, and think you ack'd, but the queue never gets the ack, so it hands out the work again to a new worker after the lock expires on it. So you could mitigate that with an additional layer of a distributed KV store. But that could have the same problem.
I run a system that processes billions of events a day and we use a system very similar to what I described above (though we have a custom queue solution and a pool of redis nodes that we have some custom quorum logic around). We don't seem to be duplicating hardly any jobs (maybe a handful a week).
If you use kafka, and only use the java clients, they say you can get exactly once delivery. See https://www.confluent.io/blog/exactly-once-semantics-are-pos....
They way they do it is by controlling the client and the server and sprinkling some write-ahead-logs with logical clocks and using a formal consensus protocol (paxos) under the hood. Even with all that, I'm skeptical of the exactly once claim.