News for hackers. Stuff that matters.
Re:Don't Use an RDBMS for Messaging by thomasz on Saturday September 27, 2014 @02:47PM (#8377824)
> I've played with the idea of a hybrid system: Register jobs in Postgres, use RabbitMQ as the trigger mechanism. But this puts a burden on the processor, which now has to use two APIs.

I've had exactly the same idea for my next project. Seriously, I'm quite surprised that something like that doesn't seem to exist already.

Re:Don't Use an RDBMS for Messaging by gfodor on Saturday September 27, 2014 @03:01PM (#8377866)
I built a similar mechanism using SQS and DynamoDB instead of RabbitMQ and PostgreSQL, respectively. The nice part is it requires no servers for job management. (Just workers obviously.)

https://github.com/gfodor/skyrunner

(It's still a bit half baked, but could be useful as a starting point.)

Re:Don't Use an RDBMS for Messaging by grosskur on Saturday September 27, 2014 @03:17PM (#8377907)
Nice, thank you. I've been looking for exactly this---SQS and DynamoDB seem like a good combination.
Re:Don't Use an RDBMS for Messaging by eropple on Saturday September 27, 2014 @05:04PM (#8378205)
You may not care, but I personally avoid SQS and Dynamo for lock-in purposes. I'd rather run my own systems and have the option of moving to GCE or something later.
Re:Don't Use an RDBMS for Messaging by gfodor on Saturday September 27, 2014 @10:32PM (#8378768)
I've found in the past that moving code between different queue and job coordination systems is pretty straightforward. Operationally of course you need to learn the new infrastructure but that is unrelated to fears about vendor lock-in.
Re:Don't Use an RDBMS for Messaging by eropple on Monday September 29, 2014 @11:59AM (#8383476)
Dynamo is more of a beef than SQS. I've found that Dynamo usage tends to grow. Like a fungus. Applying sufficient fire to remove it later seems beyond many shops' ability to marshal.

(Also, SQS is really weird about latency and places pretty ugly limits on responsiveness.)

Re:Don't Use an RDBMS for Messaging by lobster_johnson on Saturday September 27, 2014 @03:26PM (#8377935)
Out of interest, what do you think the benefits of such a system would be? I'm concerned that it would only add complexity.

For my part I plan to hide the queue processing in a central app (we're all SOA), so that workers don't need to know anything about databases, serialization formats or AMQP. They would just wait on a socket to get new jobs.

Re:Don't Use an RDBMS for Messaging by thomasz on Saturday September 27, 2014 @05:33PM (#8378273)
A message queue stores, well, messages, so it is really good for events. But I also need to persist state machines that run for months. They must be queryable and so on. A db is good at this. On the other hand, I've had a very unpleasant experience with the database-as-IPC-channel and do not wish to ever go near that again. So using a rdbms to persist state and a message queue for the events seems like the conservative approach. Am I overlooking something obvious here?
Re:Don't Use an RDBMS for Messaging by lobster_johnson on Saturday September 27, 2014 @07:52PM (#8378510)
Not at all, just wondering about your thinking. Note that I think using a database for IPC is a bad idea if you're actually using tables and such to coordinate; Postgres' listen/notify IPC is much better.

That said, it has issues, as pointed out elsewhere in this thread; when you notify, all listeners wake up, which is not what you want with a one-job-per-worker dispatch system.

If you abstracted the listening to a central job manager app, however, you could use "listen" to wake up just the job manager, and then use that wake-up to dispatch the next job to an available worker.