I'm not an expert on exchanges but my understanding is that the transaction is executed if the two sides of the order book match or the bid exceeds the ask. If you allow for a very large number of digits in the price, you reduce the probability there will be a match because of a tiny difference between the bid and ask. Won't be a problem on the most liquid securities but more of a problem on less liquid ones.
My understanding is that all prices on exchanges have a fixed number of decimal (and sometimes are even based on less granular base, like 1/8th of a decimal).
If the bid exceeds the ask there will be a trade; if not, the order will rest on the book. That's entirely normal, and a smoothly-functioning exchange should allow granular enough pricing that there would be a clear spread - when the spread is the minimum increment possible (as it usually is in equities markets - the offer is almost always just $.01 above the spread) that is a sign that there are trades that could be executing but aren't (e.g. if the bid is at 5.23 and the offer at 5.24, maybe both parties would be willing to trade at 5.233).
There's an argument that more granular prices make it harder for humans to read the order book but I don't put much faith in it.
How so? If there's a resting offer at 11.58726 and I put in a buy order, limit 11.63492, there's no reason an exchange can't match that and execute either at the midpoint or at 11.58726 +/- maker/taker (whichever the exchange policy is). What's the problem?