1. Back when scroll bars were fixed width and consistent across all applications, I would instinctively tune them out when I wasn't using them; I don't need the application to do that for me.
2. More importantly, many websites are adding some kind of progress indicator on the page itself (perhaps because the scroll bars are no longer obvious enough?). These are way more in-your-face than the scroll bars of yore.
If there were fewer elements _total_, okay I'd agree with you. But all the elements are there, just some of them are hidden, so they take _more_ time to remember how to reveal each one.
When they're all visible, you can use simple 2-D memory, and "reach for" the area that a given control is in, even before you've found it, because you know it's in that general area. Get the mouse moving there, find it, zero in on it.
When some of them are hidden, you have to remember which hiding place they're in (object permanence), then wiggle the mouse over in that area, wait for them to reveal themselves, then find the one you want, and start another motion to hit it.
Try me.
Your vision probably is correct in most of the cases as people just misuse this, yet this makes another kind of sense also: the more elements you see the harder tha is to digest for your brain (attention). Screen space has became more abundant but human attention capacity has not (perhaps it even decreased).