It's true that no matter what symbolic representation format you choose (binary or otherwise) it will never be able to encode all irrational numbers, because there are uncountably many of them.
But it's certainly false that computers can only represent rational numbers. Sure, there are certain conventional formats that can only represent rational numbers (e.g. IEEE-754 floating point) but it's easy to come up with other formats that can represent irrationals as well. For instance, the Unicode string "√5" is representable as 4 UTF-8 bytes and unambiguously denotes a particular irrational.
> representable in a finite number of digits or bits
Implying a digit-based representation.
As the other person pointed out, this is representing an irrational number unambiguously in a finite number of bits (8 bits in a byte). I fail to see how your original statement was careful :)
> representable in a finite number of digits or bits
I can define sqrt(5) in a hard-coded table on a maths program using a few bytes, as well as all the rules for manipulating it in order to end up with correct results.
Of course if you know that you want the square root of five a priori then you can store it in zero bits in the representation where everything represents the square root of five.