News for hackers. Stuff that matters.
Re:Postgres Job Queues and Failure by MVCC by CodeWriter23 on Sunday July 09, 2017 @02:40PM (#14731043)
> storing jobs in a database allows a program to take advantage of its transactional consistency; when an operation fails and rolls back, an injected job rolls back with it

This is a false premise for using a database as a queue. The same goal can be simply achieved with a different queuing system, say SQS, by only enqueuing jobs AFTER successful transaction commit.

Re:Postgres Job Queues and Failure by MVCC by IgorPartola on Sunday July 09, 2017 @02:47PM (#14731068)
What happens when the enqueuing fails after a successful commit?
Re:Postgres Job Queues and Failure by MVCC by CodeWriter23 on Sunday July 09, 2017 @08:39PM (#14732601)
Retries.
Re:Postgres Job Queues and Failure by MVCC by IgorPartola on Monday July 10, 2017 @12:39PM (#14737485)
So you retry indefinitely? How do you keep track of your retries? Do you have a queue to get the items into the real queue?

One solution to this is two phase commits, but I am not aware of any message broker that supports that. So basically you are stuck assuming that your message broker is never going to fail, or at least not going to fail for long enough for the users to notice.

Re:Postgres Job Queues and Failure by MVCC by elmigranto on Sunday July 09, 2017 @03:47PM (#14731327)
IIRC, with Postgres, you can write to Redis directly from SQL and as part of transaction. (See `redis fdw` on google.) Perhaps, there is writable FDW driver for your queue system.

(Or maybe you can just dumpt stuff to sockets by using Postgres function written in Python, JS, Perl or w/ever you prefer.)

Re:Postgres Job Queues and Failure by MVCC by ruslan_talpa on Sunday July 09, 2017 @02:48PM (#14731072)
How do you queue that job only after the transaction committed?
Re:Postgres Job Queues and Failure by MVCC by sorkin2 on Sunday July 09, 2017 @03:13PM (#14731186)
The same way the new built-in logical replication works: logical decoding.
Re:Postgres Job Queues and Failure by MVCC by ruslan_talpa on Sunday July 09, 2017 @04:41PM (#14731569)
this means it does not work on RDS
Re:Postgres Job Queues and Failure by MVCC by sorkin2 on Sunday July 09, 2017 @08:35PM (#14732583)
Tough.
Re:Postgres Job Queues and Failure by MVCC by rosser on Sunday July 09, 2017 @11:50PM (#14733266)
This is the kind of comment that makes me wish I could burn some of my own karma to downvote more than once.
Re:Postgres Job Queues and Failure by MVCC by CodeWriter23 on Sunday July 09, 2017 @08:40PM (#14732603)
Check the return code from libpq.