Agreed. I had an additional conflict in that I am a naturally sloppy person, and the worst part of doing matrix calcs by hand is that I'd often make a calculation error due to an inability to read my own handwriting, which would lead to terrible marks again and again.
Learning numpy was godsend.
That said, "working an hour by hand on some annoying 6 x 6 matrix" is how I was taught linear algebra. I got a rare B in the course (even though I aced all the tests!) because there was so much busy work I just refused to do it all. I got literally nothing out of the course in terms of understanding (there was neither time to think nor any real direction given) and a year later I couldn't even do the work anymore. I ended up picking up Axler.