Three fundamental tricks for developers writing distributed systems

Re:Three fundamental tricks for developers writing distribut... by i0exception on Monday December 03, 2012 @06:28PM (#4868574)

Isn't this the same as a message queue? Why would you want to rewrite this using a database? Also point/trick 2 seems unnecessary if you are using 3 (idempotent jobs). By queuing job ids you now have a consistency dependency between your message queue and database.

[ Parent | on HN ]

Re:Three fundamental tricks for developers writing distribut... by pedrobelo on Monday December 03, 2012 @06:45PM (#4868639)

When using a non database-based queue you'll have to find another mechanism to make sure your operation is still atomic.

In other words you can end up in a situation where a record is inserted but the job to work on it is not enqueued, or worse - that an insert fails but the job to work on it is enqueued.

Point 2 is still necessary despite idempotency imo: lets say some value is updated to "a" and then to "b", enqueuing two jobs. If the request to update "b" runs before "a" then your receiver will end up with the wrong value. Same if the initial request to update "a" fails.

[ Parent | on HN ]

Re:Three fundamental tricks for developers writing distribut... by amalter on Monday December 03, 2012 @08:16PM (#4868993)

It seems to me that XA transactions one of those patterns that need to be rediscovered every generation. I see folks start with 0mq or Redis and hit edge cases where messages get lost.

I'd love to see a somewhat simple distributed transaction standard for http api's emerge.

There has to be a middle ground between the fiddly bits of soap's ws-reliable or full on JMS broker and ad-hoc Redis queues.

[ Parent | on HN ]

Re:Three fundamental tricks for developers writing distribut... by i0exception on Monday December 03, 2012 @06:56PM (#4868686)

Perhaps I wasn't clear. What I meant was that if you enqueue the job directly and the job is idempotent, using approach 2 will introduce dependencies like the ones you pointed out.

[ Parent | on HN ]

Re:Three fundamental tricks for developers writing distribut... by codewright on Monday December 03, 2012 @06:54PM (#4868678)

When using a message queue you're forced/expected to flush messages quickly, this is often not the case.

[ Parent | on HN ]

Re:Three fundamental tricks for developers writing distribut... by i0exception on Monday December 03, 2012 @06:58PM (#4868692)

There are lots of message queues that overcome this. A good example is Apache Kafka (http://incubator.apache.org/kafka/)

[ Parent | on HN ]

Re:Three fundamental tricks for developers writing distribut... by codewright on Monday December 03, 2012 @07:09PM (#4868739)

Actually I was considering mentioning Kafka when I wrote that comment and contrasting it with RabbitMQ.

[ Parent | on HN ]