Re:Postgres Job Queues and Failure by MVCC
by IgorPartola on Sunday July 09, 2017 @02:47PM
(#14731068)
What happens when the enqueuing fails after a successful commit?
Re:Postgres Job Queues and Failure by MVCC
by CodeWriter23 on Sunday July 09, 2017 @08:39PM
(#14732601)
Retries.
Re:Postgres Job Queues and Failure by MVCC
by IgorPartola on Monday July 10, 2017 @12:39PM
(#14737485)
So you retry indefinitely? How do you keep track of your retries? Do you have a queue to get the items into the real queue?
One solution to this is two phase commits, but I am not aware of any message broker that supports that. So basically you are stuck assuming that your message broker is never going to fail, or at least not going to fail for long enough for the users to notice.
Re:Postgres Job Queues and Failure by MVCC
by elmigranto on Sunday July 09, 2017 @03:47PM
(#14731327)
IIRC, with Postgres, you can write to Redis directly from SQL and as part of transaction. (See `redis fdw` on google.) Perhaps, there is writable FDW driver for your queue system.
(Or maybe you can just dumpt stuff to sockets by using Postgres function written in Python, JS, Perl or w/ever you prefer.)
Re:Postgres Job Queues and Failure by MVCC
by ruslan_talpa on Sunday July 09, 2017 @02:48PM
(#14731072)
How do you queue that job only after the transaction committed?
The same way the new built-in logical replication works: logical decoding.
Re:Postgres Job Queues and Failure by MVCC
by ruslan_talpa on Sunday July 09, 2017 @04:41PM
(#14731569)
this means it does not work on RDS
Tough.
This is the kind of comment that makes me wish I could burn some of my own karma to downvote more than once.
Re:Postgres Job Queues and Failure by MVCC
by CodeWriter23 on Sunday July 09, 2017 @08:40PM
(#14732603)
Check the return code from libpq.
This is a false premise for using a database as a queue. The same goal can be simply achieved with a different queuing system, say SQS, by only enqueuing jobs AFTER successful transaction commit.