News for hackers. Stuff that matters.
Re:Choose Postgres Queue Technology by riku_iki on Sunday September 24, 2023 @06:30PM (#37637793)
article says he also uses "order by" clause, but I am wondering if it will severely limit throughput since all messages will need to be sorted on each lookup, but this probably can be solved by introducing index.
Re:Choose Postgres Queue Technology by vore on Sunday September 24, 2023 @07:38PM (#37638197)
It seems strictly worse to use ORDER BY in this case, since if you're using SKIP LOCKED you should be doing parallel processing anyway, and if you're doing parallel processing, ordering is already going out the window.
Re:Choose Postgres Queue Technology by nsonha on Sunday September 24, 2023 @07:53PM (#37638277)
Parallel or not, the order is of importance in any queue system.
Re:Choose Postgres Queue Technology by vore on Sunday September 24, 2023 @09:48PM (#37638868)
You have no ordering guarantees, so how can order be important? If 4 work items are scheduled on 4 independent workers, you have no guarantee which will start first or finish first.
Re:Choose Postgres Queue Technology by nsonha on Sunday September 24, 2023 @10:33PM (#37639080)
The order matters in the sense that the 5th jobs should not be atempted before those 4.
Re:Choose Postgres Queue Technology by riku_iki on Monday September 25, 2023 @12:49AM (#37639768)
I think the order matter at least because you want to have some FIFO approximation, otherwise some tasks can forever stuck in queue and never be picked up.
Re:Choose Postgres Queue Technology by vore on Monday September 25, 2023 @03:07AM (#37640467)
Then I think what you actually care about is scheduling fairness, and a strict ordering of execution of job 5 after job 4 is unimportant.
Re:Choose Postgres Queue Technology by sarchertech on Sunday September 24, 2023 @09:52PM (#37638891)
Unless you can guarantee that the processing time of each job is exactly the same, if you have multiple workers processing the same queue, you can’t order anything except the start time.

You can use locks to effectively break the queue into sub queues so that each sub queue is only being processed by 1 worker. Then you can order that sub queue.

Re:Choose Postgres Queue Technology by nsonha on Sunday September 24, 2023 @10:37PM (#37639106)
job should be attempted inthe same order/priority they are enqueued, that's the meaning of the word "queue". That they take varrying amounts of time is another matter.
Re:Choose Postgres Queue Technology by worthless-trash on Sunday September 24, 2023 @11:29PM (#37639404)
Queue can clearly mean "work that needs to be completed" not necessarily 'work completed in order'. Your definition is much stricter than it needs to be for most use cases.
Re:Choose Postgres Queue Technology by nsonha on Monday September 25, 2023 @04:29AM (#37640908)
There is clearly a conceptual difference between a set of things from which you pull things out randomly, and a queue. A queue always has intrinsic criteria to select the next item to be pulled out.
Re:Choose Postgres Queue Technology by sarchertech on Monday September 25, 2023 @08:07AM (#37642421)
There are many times when the start order doesn’t really matter, and the additional sorting overhead isn’t worth it. In those cases people will still tend to refer to the entity holding the jobs to be processed as a queue despite the fact that it doesn’t strictly follow FIFO order.

If they are being technically precise, queue isn’t the correct term, but language changes with context and time. Either way the implementation isn’t wrong if strict start order has been considered and isn’t important.

Re:Choose Postgres Queue Technology by nsonha on Monday September 25, 2023 @09:27PM (#37653165)
you're confusing between "i don't care about order", and "there is no order". Name ONE queue implementation that doesn't have order.
Re:Choose Postgres Queue Technology by worthless-trash on Tuesday September 26, 2023 @12:37AM (#37654444)
Having and caring about are different things.

I care about money, I dont have money.

Re:Choose Postgres Queue Technology by sarchertech on Tuesday September 26, 2023 @08:10AM (#37658095)
Here you go. One of the first tutorials explaining how SKIP LOCKED works in Postgres implants a job “queue” that doesn’t have an order by clause. https://www.pgcasts.com/episodes/the-skip-locked-feature-in-...

I’m not confusing anything. I’ve seen random selection “job queues” implemented many times. As long as you truly don’t care about start order, it’s fine to trade it for increased throughout.

Re:Choose Postgres Queue Technology by nsonha on Tuesday September 26, 2023 @09:42PM (#37668691)
> doesn’t have an “order by” clause

Does that mean it doesn't have any order or that you don't care about order?

Also we are arguing over whether pg suffices as a queue implementation, and you use itself as an exhibit?

Re:Choose Postgres Queue Technology by sarchertech on Wednesday September 27, 2023 @02:49PM (#37679385)
It means that you are telling pg that you don’t care about order, so it is free to optimize the query in whatever way it wants to. The order can change query to query depending on numerous external factors.

I’m not using pg itself as an example. I’m using a specific implementation of a “job queue” built with pg.

I’ve seen and you can search for and find many implementations of “job queues” using relational databases where job start order guarantees are traded away for throughput.

Re:Choose Postgres Queue Technology by CyberDildonics on Monday September 25, 2023 @09:09AM (#37643179)
not necessarily 'work completed in order'

That's exactly what a queue means, not just in every day life, but specifically in computer science.

Re:Choose Postgres Queue Technology by MaxBarraclough on Monday September 25, 2023 @03:50PM (#37649606)
This depends if we consider a priority queue to be a type of queue.
Re:Choose Postgres Queue Technology by CyberDildonics on Monday September 25, 2023 @03:57PM (#37649700)
No it doesn't. A queue always has an order. A priority queue just means you aren't always inserting at the last place in the queue.
Re:Choose Postgres Queue Technology by worthless-trash on Tuesday September 26, 2023 @08:36AM (#37658385)
From the dictionary.

Queue: a list of data items, commands, etc., stored so as to be retrievable in a definite order, usually the order of insertion.

note the term "Usually", not "always".

Re:Choose Postgres Queue Technology by nsonha on Wednesday September 27, 2023 @12:36AM (#37670202)
> to be retrievable in a definite order, usually the order of insertion.

always has an oder, which is usually of insertion.