But how is it different if you're a space hermit?

I suppose because the cost to reach you is greatly increased, meaning the cost to exercise authority is greatly increased, so actual authority is greatly deceased.

Right now the cost for the government to send their agents to "the mountains" is trivial, the most remote you could possibly get is "a few hours commute and a helicopter ride". The government is more than willing to pay that price in important cases.

I think they imagine that space travel can increase that price to "months or years" and "expensive vehicles that support people during that time". Rather like how much it would have cost to send government agents to remote places hundreds of years ago.

(Personally I'm not particularly convinced, the number of habitable rocks in the solar system is small. Living not on a rock means you need to import resources. Maybe if we get interstellar travel).