News for hackers. Stuff that matters.
Re:Random Numbers in Bash by gorgoiler on Monday March 27, 2023 @10:41AM (#35327285)
The go-to source for randomness in Linux is /dev/urandom, is it not?

I’m not sure the best way to make use of it as a source of random decimals — there are surely a dozen ways to do so — but this works:

  printf %d 0x`
    head -c 1000 /dev/urandom
    | shasum 
    | head -c 8
  `
As always, my advice for things like this is that if you’re in need of true random numbers in your shell script then you should probably stop writing whatever it is you are writing as a shell script.
Re:Random Numbers in Bash by masklinn on Monday March 27, 2023 @11:41AM (#35328127)
> The go-to source for randomness in Linux is /dev/urandom, is it not?

AFAIK it's the true source of randomness in any unix.

> I’m not sure the best way to make use of it as a source of random decimals — there are surely a dozen ways to do so — but this works:

`od(1)` will do binary to binary to decimal:

    > head -c4 /dev/urandom | od -An -vtu4
               4119929390                                                

    > head -c4 /dev/urandom | od -An -vtu4
               3540752584                                                

    > head -c4 /dev/urandom | od -An -vtu4
                998126897
Re:Random Numbers in Bash by ary on Monday March 27, 2023 @11:51AM (#35328230)
> As always, my advice for things like this is that if you’re in need of true random numbers in your shell script then you should probably stop writing whatever it is you are writing as a shell script.

The use of "true" here seems ambiguous.

Does this statement mean that you discourage creation of cryptographic applications with shell scripts, you discourage use of shell scripts to orchestrate applications that use /dev/urandom for any reason, or something else entirely?

Re:Random Numbers in Bash by noam_k on Monday March 27, 2023 @12:02PM (#35328363)
"True random" as opposed to "pseudo random", which is not cryptographically secure.
Re:Random Numbers in Bash by jimktrains2 on Monday March 27, 2023 @12:31PM (#35328753)
There are cryptographically secure pseudorandom generators. In fact all sources non-hardware (i.e. /dev/random and /dev/urandom) are psuedo random as are the keystream generators for most algos that use one.
Re:Random Numbers in Bash by noam_k on Monday March 27, 2023 @01:05PM (#35329221)
TIL. Thanks!
Re:Random Numbers in Bash by ary on Monday March 27, 2023 @01:15PM (#35329379)
Yes, I'm aware of that. The commenter seems to be implying that one should stop using a shell script when one needs "true" random, and I'm curious as to why since one can call out to Perl, Python, etc.

The linked gist doesn't appear to be making the case for using its recommendations in cryptographic applications so what I'm ultimately after is what the commenter thinks is being done in the shell that's inappropriate.

Re:Random Numbers in Bash by MattPalmer1086 on Monday March 27, 2023 @01:41PM (#35329785)
Incorrect. I'm not sure what you mean by "true random", and we could get into some interesting philosophical discussion on that.

However, all software random number generators are pseudo random, including cryptographic ones. They are called CSPRNGs, which stands for Cryptographically Secure Pseudo Random Number Generator.

Re:Random Numbers in Bash by chasil on Monday March 27, 2023 @11:54AM (#35328273)
There are several scenarios where a cryptographic-quality random number generator might be needed within the context of a shell script.

My favorite tool for this is "openssl rand" and I can find the manual page on rhel7 with "man rand."

Here are 8 bytes of randomness in base64:

  $ openssl rand -base64 8 
  mGHxYcVOmA8=
When I've got dozens of users to create, this is how I am assigning their initial passwords.

The shell's $RANDOM is somewhat less trustworthy than other sources here. It also does not appear to be POSIX, but comes from ksh88, where it is described as returning a random integer between 0 and 32767 each time it is referenced. If unset in a script, it cannot be used again for the life of the shell.

Re:Random Numbers in Bash by TRiG_Ireland on Monday March 27, 2023 @12:46PM (#35328962)
I use this to create passwords when I need to manually:

    function password_generate { head -c ${1-12} /dev/urandom | base64; }
Re:Random Numbers in Bash by chasil on Monday March 27, 2023 @01:27PM (#35329565)
Sometimes, extreme portability makes us do desperate things.

  # uname -a
  HP-UX prdcrm B.10.20 A 9000/800 862741461 two-user license

  # ls -l /dev/urandom
  /dev/urandom not found

  # openssl rand -base64 8
  v51U1RvSD2Q=
I could run this on VAX VMS if you want.
Re:Random Numbers in Bash by rascul on Monday March 27, 2023 @12:07PM (#35328421)
If I recall correctly, /dev/urandom and /dev/random work the same now.
Re:Random Numbers in Bash by dharmab on Monday March 27, 2023 @01:56PM (#35329965)
That change was reverted, and I do not know if they were re-unified since.

https://lwn.net/Articles/889452/

Pragmatically, they are equivalent for most scripting use cases.

Re:Random Numbers in Bash by tzs on Monday March 27, 2023 @12:11PM (#35328478)
Set RD to $N random bytes in decimal:

  RD=$(dc -e 16i`xxd -c $N -g $N -l $N -p /dev/urandom | tr a-f A-F`p)
Re:Random Numbers in Bash by radiator on Monday March 27, 2023 @12:42PM (#35328901)
Unfortunately, this is neither readable nor particularly concise.
Re:Random Numbers in Bash by hayst4ck on Monday March 27, 2023 @12:13PM (#35328509)
While being overly pedantic (because "not true" random is a spectrum), it is good practice to use random numbers to ensure that a recurring cron job does not run at the same time on all machines.

This goes from good practice to necessary as a cron job uses increasing resources or talks to a centralized service.

Puppet implements this as splay: https://www.puppet.com/docs/puppet/7/configuration.html#spla...

Re:Random Numbers in Bash by dharmab on Monday March 27, 2023 @01:54PM (#35329933)
Systemd offers multiple options for randomized delays for timers, documented at: https://www.freedesktop.org/software/systemd/man/systemd.tim...
Re:Random Numbers in Bash by kevin_thibedeau on Monday March 27, 2023 @02:19PM (#35330232)
You don't need the shell for that. Every modern cron supports randomization internally.
Re:Random Numbers in Bash by wodenokoto on Monday March 27, 2023 @10:31AM (#35327135)
Seems odd that these convenience function don't draw from either /dev/random or /dev/urandom/
Re:Random Numbers in Bash by geraldcombs on Monday March 27, 2023 @02:01PM (#35330037)
I'm pretty sure $RANDOM predates /dev/{u,}random on most OSes.
Re:Random Numbers in Bash by Tepix on Monday March 27, 2023 @10:39AM (#35327246)
I use "openssl rand X" where X is the number of pseudo-random bytes i need.

Say you want to generate a strong password, you can use this handy function that works in bash and zsh:

    function gen_passwd() {
       openssl rand 240 |\
       LC_CTYPE=C tr -dc '[:graph:]' |\
       cut -c 1-${1:-20}
    }
Re:Random Numbers in Bash by Karellen on Monday March 27, 2023 @11:22AM (#35327878)
If you have a password manager installed on your system (and you should) then it might have a command-line client that can generate new passwords. e.g.:

    karellen@localhost:~$ keepassxc-cli generate -lU
    hpJUwcYZnzJyAbrnvTWnTzrlqzqcdvsl
    karellen@localhost:~$ keepassxc-cli diceware
    diabetes dinginess goldfish purr cried expenses related
See `keepassxc-cli help generate` and `keepassxc-cli help diceware` for more info, or the man page: https://manpages.debian.org/bullseye/keepassxc/keepassxc-cli...
Re:Random Numbers in Bash by zikduruqe on Monday March 27, 2023 @12:06PM (#35328418)
Or just use what you already have on your machine...

    sort -R /usr/share/dict/words | head -n 4| sed 's/.*/&/;$!s/$// ' |tr '\n' '-' |sed 's/-$/\n/'
Re:Random Numbers in Bash by r3trohack3r on Monday March 27, 2023 @12:08PM (#35328434)
Oh handy - didn’t think of using that word list. I download the EFF diceware list.
Re:Random Numbers in Bash by zikduruqe on Monday March 27, 2023 @01:18PM (#35329410)

    sort -R /usr/local/share/eff_short_wordlist_1.txt |awk '{print$2}' | head -n 4| tr '\n' '-' | sed 's/-$/\n/'
Re:Random Numbers in Bash by Karellen on Monday March 27, 2023 @01:03PM (#35329199)
Be careful with that:

> The shuf, shred, and sort commands sometimes need random data to do their work. For example, ‘sort -R’ must choose a hash function at random, and it needs random data to make this selection.

> By default these commands use an internal pseudo-random generator initialized by a small amount of entropy, but can be directed to use an external source with the --random-source=file option. An error is reported if file does not contain enough bytes.

-- https://www.gnu.org/software/coreutils/manual/html_node/Rand...

(emphasis mine)

You might end up with a fairly easy-to-brute-force password using that method.

They say "never roll your own crypto", but I think that advice probably holds pretty well for password generators(/managers). I'll stick with using a dedicated program. I reckon it's far less likely to have obvious blunders than something that's just been cobbled together using whatever tools happened to be lying around nearby.

Re:Random Numbers in Bash by r3trohack3r on Monday March 27, 2023 @12:07PM (#35328430)
Personally I have a tiny shell function that uses openssl rand with a dice ware list. It accepts a length, and it returns CamelCase.

I use it for usernames since Bitwarden doesn’t support diceware names.

Re:Random Numbers in Bash by rlpb on Monday March 27, 2023 @12:33PM (#35328776)
pwgen is great as a password generator, and is a tiny package.
Re:Random Numbers in Bash by gavinhoward on Monday March 27, 2023 @10:36AM (#35327209)
Shameless plug: my bc [1] has a random number generator that is much higher quality than the shell one. Because it's bc, you can generate random numbers of any size, and you can generate random reals as well.

It is also seeded.

[1]: https://git.gavinhoward.com/gavin/bc

Re:Random Numbers in Bash by queuebert on Monday March 27, 2023 @01:10PM (#35329294)
Can you give an example usage? I cannot figure out how to invoke your PRNG.
Re:Random Numbers in Bash by gavinhoward on Monday March 27, 2023 @02:01PM (#35330033)
If you want just a number to store in a hardware integer type, use this:

    echo "rand()" | bc
If you want a bounded integer, use this:

    echo "irand(<bound>)" | bc
This bound is adjusted to not bias the random numbers.

It can also be used to generate either really small or really large integers, arbitrary size. My bc will take care of everything behind the scenes.

If you want to generate a real between 0 and 1, use this:

     echo "frand(<places>)" | bc -l
where <places> is the number of decimal places you want. Note the "-l" argument; this function is not built-in like the other two.

If you want an arbitrary real (not between 0 and 1), use this:

    echo "ifrand(<bound>, <places>)" | bc -l
My bc will seed itself from the system CSPRNG, but you can seed the PRNG by assigning the seed to the special variable `seed` first:

    echo "seed = $SEED; rand()" | bc
You can even save the current seed by getting the value of the `seed` variable:

    echo "seed" | bc
If you want to use the same sequence over multiple invocations of bc, you need to save the seed to pass to the next invocation:

    OUT=$(echo "rand(); seed" | bc)
    RAND=$(echo "$OUT" | head -n1)
    SEED=$(echo "$OUT" | tail -n1)
    RAND2=$(echo "seed = $SEED; rand()" | bc)
(Haven't tried the above code. Sorry if it's wrong.)

This pattern works because the `seed` variable is updated after every call to the PRNG.

Does that help?

Re:Random Numbers in Bash by queuebert on Tuesday March 28, 2023 @01:12PM (#35344420)
Thank you, this is perfect!
Re:Random Numbers in Bash by ape4 on Monday March 27, 2023 @10:33AM (#35327169)
There's always `head -c20 /dev/random | base64`
Re:Random Numbers in Bash by xxs on Monday March 27, 2023 @11:12AM (#35327742)
likely urandom but the point still stands.
Re:Random Numbers in Bash by frutiger on Monday March 27, 2023 @10:24AM (#35327038)
From the conclusion of the OP:

> I could do more experiments and try to definitively determine what's happening, bottom line, don't use $RANDOM, even for unimportant random numbers.

Re:Random Numbers in Bash by alyandon on Monday March 27, 2023 @10:38AM (#35327239)
As hexdump tends to always exist (vs python) I've used the following before:

  hexdump -e '"%d"' -n2 /dev/urandom
Re:Random Numbers in Bash by torcete on Monday March 27, 2023 @10:46AM (#35327350)
I assume that the random numbers in bash come from an uniform distribution. Is there a way of getting random numbers from a different distribution with given parameters?

I can only think of one-liners in R or maybe python.

Re:Random Numbers in Bash by zamadatix on Monday March 27, 2023 @10:55AM (#35327495)
The standard approach is to take the CDF form of the desired distribution, solve it for equaling some constant, then plug in your uniform value and out comes where on the desired distribution that maps to. The hard part is you either have to pre-solve it or use numerical solving methods and in either case you'd have to write that in Bash.
Re:Random Numbers in Bash by pmarreck on Monday March 27, 2023 @11:55AM (#35328289)
for a normal distribution, chatgpt suggests the box-muller transform and calling out to awk (which is a scripting language that continues to surprise me; it seems underrated?)

    normal_random() {
        start=$1
        end=$2
        range=$(echo "$end - $start" | bc -l)

        awk -v start=$start -v range=$range -v seed=$RANDOM '
        BEGIN {
            srand(seed);
            u1 = rand();
            u2 = rand();
            z0 = sqrt(-2 * log(u1)) * cos(2 * 3.14159265358979323846 * u2);
            z1 = sqrt(-2 * log(u1)) * sin(2 * 3.14159265358979323846 * u2);

            random_number = start + (z0 * (range / 6)) + (range / 2);
            printf("%.0f\n", random_number);
        }'
    }
In testing it out, they did seem normally-distributed but rerunning it quickly did seem to produce more consecutively duplicate results than would be expected, not sure why that would be (I was using ranges of 0-10 and 0-20, perhaps the probability of normally-distributed consecutive dupes in that range is quite higher?)

EDIT: Nope, there's definitely something wrong with this, possibly the rand() call(s)? Can anyone guess? (A real-world example of the perils of relying on chatGPT!) EDIT2: Figured it out. srand with no args seeds to the current second, making any run of this function within the same second produce the same result. Fixing this left as an exercise to the reader EDIT3: I fixed it by seeding it externally via $RANDOM.

Re:Random Numbers in Bash by jwilk on Monday March 27, 2023 @01:41PM (#35329791)
z1 is never used.
Re:Random Numbers in Bash by pmarreck on Monday March 27, 2023 @02:34PM (#35330462)
Good point! chatGPT explained it thusly:

"The Box-Muller transform generates two normally distributed random numbers, z0 and z1, using the two uniformly distributed random numbers u1 and u2. However, only z0 is used in the provided function to create the random number within the given range. If you want to additionally supply z1 in the output, you can." It then provides the modified code.

Re:Random Numbers in Bash by sureglymop on Monday March 27, 2023 @05:18PM (#35332786)
AWK is amazing. I recommend reading "The AWK Programming Language", read it as a kid! You can even do network programming with gawk: https://www.gnu.org/software/gawk/manual/html_node/TCP_002fI...
Re:Random Numbers in Bash by nixcraft on Monday March 27, 2023 @11:03AM (#35327615)
Are you on GNU/Bash version 5.1[0]? A new bash variable, SRANDOM added to GNU/Bash-5.1 release, which gets its random data from the system’s entropy engine and is not lined and cannot be reseeded to get an identical random sequence. For example:

   echo "$SRANDOM"
   for r in {1..5}; do printf "%s\n" "$SRANDOM"; done 
[0] https://www.cyberciti.biz/linux-news/gnu-bash-5-1-released-w...
Re:Random Numbers in Bash by tux on Monday March 27, 2023 @11:14AM (#35327763)
Or something like this; echo -e "$SRANDOM"{1..5}"\n"
Re:Random Numbers in Bash by jwilk on Monday March 27, 2023 @11:34AM (#35328032)
Er, I don't think you want that.
Re:Random Numbers in Bash by jwilk on Monday March 27, 2023 @01:27PM (#35329563)
Do you really wanted to append digits 1, 2, 3, 4, 5 to your high-quality random numbers?
Re:Random Numbers in Bash by tux on Monday March 27, 2023 @11:04AM (#35327627)
You can also use "shuf" to create random listen port for aria2c.

PORT=$(shuf -i 1000-2000 -n1)

aria2c --listen-port=$PORT ...other...options...

Re:Random Numbers in Bash by jwilk on Monday March 27, 2023 @11:29AM (#35327970)
Fun fact: bash's $RANDOM never yields the same value twice in a row:

  $ for i in {1..999999}; do echo $((RANDOM-RANDOM)); done | sort -nu | grep -E '^-?.$'
  -9
  -8
  -7
  -6
  -5
  -4
  -3
  -2
  -1
  1
  2
  3
  4
  5
  6
  7
  8
  9
Re:Random Numbers in Bash by mywittyname on Monday March 27, 2023 @02:57PM (#35330788)
Maths isn't my background, but isn't this a serious flaw with the quality of the pseudo-random numbers it generates?

I remember learning about the enigma machine and how it had a flaw where a letter could not be transformed into itself. And knowing that, for example, an A could never be a A was critical to cracking the enigma code. Because of that knowledge, my cryptographic intuition is that any seemingly minor fact like this could actually be a critical flaw.

Re:Random Numbers in Bash by jwilk on Monday March 27, 2023 @05:45PM (#35333117)
I doubt it matters much. Bash's $RANDOM is implemented using a linear congruential generator, which is not suitable for cryptographic applications anyway.
Re:Random Numbers in Bash by pmarreck on Monday March 27, 2023 @11:46AM (#35328180)
Anyone want to critique this deterministic random number generator I wrote not long ago in Bash that takes advantage of sha256 sums?

https://gist.github.com/pmarreck/e65f457e8755e461a87db7c94d4...

Re:Random Numbers in Bash by sltkr on Monday March 27, 2023 @12:19PM (#35328593)
$((32#$DRANDOM_SEED)) is not correct: the sha256 hash is in base 16, not 32. With your current implementation every fifth bit is always 0.
Re:Random Numbers in Bash by pmarreck on Monday March 27, 2023 @12:30PM (#35328738)
Ooooh, good find!
Re:Random Numbers in Bash by VLM on Monday March 27, 2023 @12:02PM (#35328366)
The problem with random as a word is it covers user intents ranging from "crudely load balance well enough most of the time" or "add a small variable delay to smooth the load over time" up to extremely security critical crypto applications.
Re:Random Numbers in Bash by dennisse on Monday March 27, 2023 @01:13PM (#35329350)
Since no-one asked, I use something like this in zsh:

  rand() {
    local r4
    IFS= read -rk4 -u0 r4 < /dev/urandom || return
    local b1=$r4[1] b2=$r4[2] b3=$r4[3] b4=$r4[4]
    print $(( #b1 << 24 | #b2 << 16 | #b3 << 8 | #b4 ))
  }
It has the added bonus of only using zsh, no external dependencies, so it is quite fast.
Re:Random Numbers in Bash by mg on Monday March 27, 2023 @02:13PM (#35330166)
I used this to create random decisions for a while:

    if (( RANDOM % 2 == 0 )); then echo Yes; else echo No; fi
I also logged those decisions. I got significantly more No then Yes. I could never reproduce the bias in a loop. But in real life usage, I got more No. Could be by chance of course. But it made me worried enough that I switched to urandom:

    bit=$(od -An -N1 -i /dev/urandom)
    if (( $bit &1 ))
        then echo "Yes"
        else echo "No"
    fi
Re:Random Numbers in Bash by patrakov on Monday March 27, 2023 @03:06PM (#35330920)
Another fun fact: on Linux systems, the /proc/sys/kernel/random/uuid file contains a random UUID. This can be used to get 32-bit random numbers in POSIX shell without calling out to external utilities. Quite handy on OpenWrt.

read -r < /proc/sys/kernel/random/uuid

MYRAND=$(( 0x${REPLY%%-*} ))

Re:Random Numbers in Bash by qazxcvbnm on Tuesday March 28, 2023 @08:55AM (#35340194)
On a related note, I've once had to reproducibly generate a random sequence of N-bytes, and one of the more elegant ways I discovered and ended up using was to 'encrypt' a known sequence of bytes with the required length (so that the sequence will be long enough) with some known 'password', and truncating it.

`( yes '00' | head -n 100000 | tr -d $'\n') | xxd -r -p | aespipe -e AES256 -P <(echo 'your-seed')`