I had the same objection as the op actually - if you wake up at "rest" with just yourself and a bed, and accelerated away from the bed, you can in fact know whether "you are moving" or "the bed is moving". It is because one of the two have only been in 1 reference frame, while the other has been in 2.

Basically the same issue as the twin "paradox". I can appreciate the assumption of a constant velocity on wake though, as a way to simplify the story.