Hardest thing about pointers is that it forces the programmer to switch abstraction levels. One moment you're thinking about values and the next your thinking about memory addresses and the values at those addresses. Also, I've gotten more than one :| look when I've said to people that pointers are variables of their own and you can get pointers to them and do all sorts of nasty stuff.

I have a feeling that if you learned Assembler and then a language without pointers and then came to C, you'd still be confused.