Why Adaptive Rate Limiting Is a Game-Changer

Re:Why Adaptive Rate Limiting Is a Game-Changer by michaelcampbell on Saturday July 22, 2023 @03:08PM (#36828880)

Not sure it "changed my game" but I have a process that I run on many files at once. In a *nix I would run the processes like so:

    for i in list_of_files; do
      run_intensive_process &
      sleep $(awk '{print $3}' < /proc/loadavg)
    done

As more processes start upping my load average, it sleeps longer before running each one in the background. I guess GNU parallel can do this sort of thing with a load-sensitive command line arg, but I find this pattern easy to remember, and parallel's decidedly not.

Works for me.

[ Parent | on HN ]

Re:Why Adaptive Rate Limiting Is a Game-Changer by twic on Saturday July 22, 2023 @04:02PM (#36829312)

You could also hand all the invocations of run_intensive_process to at, and let it take care of running them when the load average permits:

https://linux.die.net/man/1/at

I don't think at is very heavily used on modern systems, so it's quite possible that it doesn't actually have very useful behaviour here.

[ Parent | on HN ]

Re:Why Adaptive Rate Limiting Is a Game-Changer by dredmorbius on Sunday July 23, 2023 @04:27AM (#36833548)

More specifically, batch(1).

at(1) runs jobs after a specified time. (A common mistake is to presume that "at" means "will run when specified" when in fact it means "will run no earlier than specified", and depending on system characteristics, could run substantially later.)

batch runs jobs ... as system load permits, without any time specification, in a FIFO queue.

There's also cron, the third Unix scheduler, which executes a set of specified tasks on a regular schedule. All three are related.

[ Parent | on HN ]

Re:Why Adaptive Rate Limiting Is a Game-Changer by twic on Sunday July 23, 2023 @08:28AM (#36834711)

I have the idea that at is more widespread than batch, and since you can say 'at now' (or 'at $(date)'), at is a better bet.

As far as I know, cron doesn't pay attention to load average, but I would love to be corrected.

All of this feels like a very crufty and highly neglected part of Unix, which is a shame. There's an alternate history of Unix where it's all about batch jobs and datagram Unix domain sockets.

[ Parent | on HN ]

Re:Why Adaptive Rate Limiting Is a Game-Changer by michaelcampbell on Sunday July 23, 2023 @03:01PM (#36838535)

> As far as I know, cron doesn't pay attention to load average, but I would love to be corrected.

It does not. Up to whatever you run from cron to deal with that.

[ Parent | on HN ]

Re:Why Adaptive Rate Limiting Is a Game-Changer by dredmorbius on Sunday July 23, 2023 @03:10PM (#36838626)

at and batch are both part of the same underlying facility. The former seems more known than batch, but where you find one you will pretty much always find the other.

On many systems they'll share an integrated manpage.

[ Parent | on HN ]

Re:Why Adaptive Rate Limiting Is a Game-Changer by caro11ne on Monday July 24, 2023 @02:01AM (#36843615)

Stop starting more jobs if load > number of cpus:

    cat list_of_file | parallel --load 100% run_intensive_process

[ Parent | on HN ]