Octobot - A Low-Latency, Highly Parallel Distributed Task Queue Worker

Re:Octobot - A Low-Latency, Highly Parallel Distributed Task... by cnlwsu on Monday September 20, 2010 @03:08PM (#1710102)

From my personal experience with these, Hazelcast seems to be better suited for something like a single worker queue... We had issues when we had thousands of distributed queues on different systems. We ended up going with Terracotta (which handles this well if you work through the lock contention).

This project sounds promising for that kind of application but wont know until I try it out.

[ Parent | on HN ]

Re:Octobot - A Low-Latency, Highly Parallel Distributed Task... by DEinspanjer on Monday September 20, 2010 @03:55PM (#1710297)

Sorry for my confusion, so is the problem you experienced one of having thousands of queues that a small number of clients pulled from, or thousands of workers that worked off of a small number of queues?

In our case, we want to be having a very small number of queues that have a high rate of i/o with dozens of reader clients and writer clients.

[ Parent | on HN ]

Re:Octobot - A Low-Latency, Highly Parallel Distributed Task... by bananaandapple on Monday September 20, 2010 @06:20PM (#1710701)

Could you eloborate where you did have problems? We also plan on using a small number of queues having a high rate of readers and clients (e.g. distributed task execution).

[ Parent | on HN ]

Re:Octobot - A Low-Latency, Highly Parallel Distributed Task... by cnlwsu on Monday September 20, 2010 @11:08PM (#1711329)

We have a large number of queues with small number of workers.

[ Parent | on HN ]