It could be bigger. The 'keys' are just arbitrary sytems of equations, this is where 'unbounded keyspace' comes from. Any implementation would wind up being bounded by some practical constraint.

Whether this is any improvement on existing ciphers (which you can also design to have any size key you desire), is less clear, but I would bet on 'no'.

It's typical for crypto algorithms to generate an endless (or extremely large) "keystream" by permutating a much shorter secret key with other data somehow.

The fact they're not using the term "keystream" for this concept is a bit of a warning that they might've reinvented something every cipher algo already does, but more inefficient (from skimming through the paper).

Just guessing though. Not judging. I'm not a cryptographer, I just read about crypto as a hobby (and to know what I'm doing while using someone else's crypto primitives in my code).