Sequential and parallel execution of long-running shell commands

Re:Sequential and parallel execution of long-running shell c... by veyh on Sunday March 24, 2024 @02:48PM (#39809254)

Pueue dumps the state of the queue to the disk as JSON every time the state changes, so when you have a lot of queued jobs this results in considerable disk io. I actually changed it to compress the state file via zstd which helped quite a bit but then eventually just moved on to running NATS [1] locally.

[1] https://nats.io/

[ Parent | on HN ]

Re:Sequential and parallel execution of long-running shell c... by Nukesor on Monday March 25, 2024 @07:46PM (#39822588)

Interesting.

May I ask how many tasks you were managing with Pueue and what your usecase was?

I also thought about using alternative formats such as CBOR, but choosing a human-readable format like JSON made debugging and such a lot easier.

If there's a good usecase for it, I might consider switching to a more compact format.

[ Parent | on HN ]

Re:Sequential and parallel execution of long-running shell c... by veyh on Wednesday March 27, 2024 @03:47AM (#39836524)

I was running a lot of small tasks. With 3000 queued tasks my state file was around 25 MB.

zstd compressed that down to 5% of the size. I have the code still if you want to look at it but it was just a quick experiment so I didn't add any tests. I did add it to the config, disabled by default, though.

https://github.com/veyh/pueue/commit/e9dcf52227304b4b4a2ded4...

[ Parent | on HN ]