Re:Git scraping: track changes over time by scraping to a Gi...
by madphilosopher on Friday August 11, 2023 @01:24AM
(#37085381)
Further to this, I also put the following snippet in front of my web-scraping cron jobs so they start at a random time between the minute boundaries:
perl -le 'sleep rand 60';
It's the most compact code I could find to do the job.
Re:Git scraping: track changes over time by scraping to a Gi...
by drougge on Saturday August 12, 2023 @10:16AM
(#37100434)
If you're using a somewhat modern shell there is $RANDOM which gives you a 15 bit random number. So e.g.
sleep $((RANDOM / 546))
but I guess most cron jobs run with an extremely conservative shell that might not have it.
Not sure how much of a difference it makes to the underlying service but I will also do this with my scraping.
Thank you for point that out