News for hackers. Stuff that matters.
Re:Replacing cron jobs with a centralized task scheduler by diarrhea on Friday August 01, 2025 @03:08AM (#44753835)
This introduces long-running transactions, which at least in Postgres should be avoided.
Re:Replacing cron jobs with a centralized task scheduler by danielheath on Friday August 01, 2025 @04:14AM (#44754200)
Depends what else you’re running on it; it’s a little expensive, but not prohibitively so.
Re:Replacing cron jobs with a centralized task scheduler by maxbond on Friday August 01, 2025 @05:17AM (#44754578)
Long running transactions interfere with vacuuming and increase contention for locks. Everything depends on your workload but a long running transactions holding an important lock is an easy way to bring down production.
Re:Replacing cron jobs with a centralized task scheduler by iglio on Friday August 01, 2025 @06:07AM (#44754833)
If the system is already using SQS, DynamoDB has this locking library which is lighter weight for this use case

https://github.com/awslabs/amazon-dynamodb-lock-client

> The AmazonDynamoDBLockClient is a general purpose distributed locking library built on top of DynamoDB. It supports both coarse-grained and fine-grained locking.

Re:Replacing cron jobs with a centralized task scheduler by aitchnyu on Friday August 01, 2025 @06:14AM (#44754895)
I read too many "use Postgres as your queue (pgkitchensink is in beta)", now I'm learning listen/notify is a strain, and so are long transactions. Is there a happy medium?
Re:Replacing cron jobs with a centralized task scheduler by bmn__ on Friday August 01, 2025 @08:40AM (#44755911)
Just stop worrying and use it. If and when you actually bump into the limitations, then it's time to sit down and think and find a supplement or replacement for the offending part.
Re:Replacing cron jobs with a centralized task scheduler by rco8786 on Friday August 01, 2025 @04:13PM (#44761923)
Excellent advice across many domains/techs here.
Re:Replacing cron jobs with a centralized task scheduler by thewisenerd on Friday August 01, 2025 @06:27AM (#44754973)
t1: select for update where status=pending, set status=processing

t2: update, set status=completed|error

these are two independent, very short transactions? or am i misunderstanding something here?

--

edit:

i think i'm not seeing what the 'transaction at start of processor' logic is; i'm thinking more of a polling logic

    while true:
      r := select for update
      if r is None:
        return
      sleep a bit
this obviously has the drawback of knowing how long to sleep for; and tasks not getting "instantly" picked up, but eh, tradeoffs.
Re:Replacing cron jobs with a centralized task scheduler by maxbond on Friday August 01, 2025 @11:28AM (#44758284)
They're proposing doing it in one transaction as a heartbeat.

> - If you find an unlocked task in 'executing', you know the processor died for sure. No heuristic needed

Re:Replacing cron jobs with a centralized task scheduler by diarrhea on Saturday August 02, 2025 @03:34AM (#44765532)
Yes, and that cannot work: if a task is unlocked but in 'executing' state, how was it unlocked but its state not updated?

If a worker/processor dies abruptly, it will neither unlock nor set the state appropriately. It won't have the opportunity. Conceptually, this failure mode can always occur (think, power loss).

If such a disruption happened, yet you later find tasks unlocked, they must have been unlocked by another system. Perhaps Postgres itself, with a killer daemon to kill long-running transactions/locks. At which point we are back to square one: the job scheduling should be robust against this in the first place.

Re:Replacing cron jobs with a centralized task scheduler by diarrhea on Saturday August 02, 2025 @03:41AM (#44765570)
Your version makes sense. I understood the OP's approach as being different.

Two (very, if indexed properly) short transactions at start and end are a good solution. One caveat is that the worker can die after t1, but before t2 - hence jobs need a timeout concept and should be idempotent for safe retrying.

This gets you "at least once" processing.

> this obviously has the drawback of knowing how long to sleep for; and tasks not getting "instantly" picked up, but eh, tradeoffs.

Right. I've had success with exponential backoff sleep. In a busy system, means sleeps remain either 0 or very short.

Another solution is Postgres LISTEN/NOTIFY: workers listen for events and PG wakes them up. On the happy path, this gets instant job pickup. This should be allowed to fail open and understood as a happy path optimization.

As delivery can fail, this gets you "at most once" processing (which is why this approach by itself it not enough to drive a persistent job queue).

A caveat with LISTEN/NOTIFY is that it doesn't scale due to locking [1].

[1]: https://www.recall.ai/blog/postgres-listen-notify-does-not-s...

Re:Replacing cron jobs with a centralized task scheduler by maxbond on Saturday August 02, 2025 @03:53AM (#44765603)
What are you thoughts on using Redis Streams or using a table instead of LISTEN/NOTIFY (either a table per topic or a table with a compound primary key that includes a topic - possibly a temporary table)?
Re:Replacing cron jobs with a centralized task scheduler by diarrhea on Saturday August 02, 2025 @04:13AM (#44765679)
I've not used Redis Streams, but it might work. I've seen folks advise against PG, in favor of Redis for job queues.

> using a table instead of LISTEN/NOTIFY

What do you mean? The job queue is backed by a PG table. You could optionally layer LISTEN/NOTIFY on top.

I've had success with a table with compound, even natural primary keys, yes. Think "(topic, user_id)". The idea is to allow for PARTITION BY should the physical tables become prohibitively large. The downsides of PARTITION BY don't apply for this use case, the upsides do (in theory - I've not actually executed on this bit!).

Per "topic", there's a set of workers which can run under different settings (e.g. number of workers to allow horizontal scaling - under k8s, this can be automatic via HorizontalPodAutoscaler and dispatching on queue depth!).