You're right. In practice I usually C-/ (undo) C-g / (redo) to get back to where I was; which is something you can do after you've lost your place in the file, but it assumes to previously made some change in that location.