A lot of mathematics people feel this way about programming, too. Both sides expect one to devote the time to build intuition to grok their subject. I think it's easier to learn programming coming from the mathematics side, because you're going from nobody holding your hand to a compiler giving you error messages. You can't interactively test whether your intuition in mathematics is sound (you can, but you have to play both the role of the compiler and the tester).
Eventually I just ragequit math because I didn't have the patience and time to search for the intuition myself, which is a pitty. I'm really happy I followed the classes about mathematical proofs, groups etc because I really enjoyed those but others were just painful.
Anyhow recently I discovered this blog too up my math concerning computer science is this: http://jeremykun.com/. It's a tad more advanced than this but I'd really recommend it. His blog is freaking amazing and always a joy to read. Math as a sidehobby it is! :)