Eh, you use rotations if you're specifically looking for rotation invariance. Also, it's hard to tell how confused the network actually is since it's not really predicting 'probability' of a class (even though the term is commonly used). Quite often neural nets just output the most 'probable' class with an oversized probability estimate due to how the most common classification layer works.