It does not need to be either/or. I have configured the terminal mode of ansi-term to allow the cursor keys to move around in the buffer as usual exactly because I often like to back up a bit to look at some output and keeping remembering that the normal buffer movement keys was not available became annoying.