If you fall asleep during a video, now you'll have no idea where you were. Similarly if I lose concentration watching a video, I now have to either go back or hope I didn't miss anything important. If I lose concentration while reading, the page is still there, I just have to skim it until I find where i went on a tangent.

You still need to skim the page. The same way you can skim video to find the place you remember. I don't see a lot of difference here.

You really don't see a difference between skimming a single page and scrubbing through a video that could be hours long? And scrubbing only works when the video is visually distinct enough between parts that you can tell without having to watch a few seconds at multiple different points.