This obviously isn't a solution to your valid complaint, but you can use C-u C-SPC to navigate to your previous cursor position. It can band-aid your issue, but it's also quite useful in general.

https://www.gnu.org/software/emacs/manual/html_node/emacs/Ma...