That shouldn't happen unless you've mapped "j" to "gj" (which is fairly common for people who use line wrapping): http://vimdoc.sf.net/htmldoc/motion.html#gj
If you?ve remapped `j` and `k` to move visually, you can still move linewise using `<CR>` (Enter) and `-`. They are equivalent to `j^` and `k^`.
Say I want to move 6 lines down. I use 6j, but one of the lines I want to jump over is wrapped and occupies 2 lines in my editor. That will mean my cursor will end up at the line that was previous numbered 5. This behavior drives me nuts.