There's a potential that someone who's currently using some app without side-loading won't have the opportunity to keep using the app without side-loading as a result of a decision to allow side-loading. That means "this decision doesn't affect you, just keep using apps without side-loading" isn't a valid statement.

You're right that everyone will technically always have the choice not to side-load. Just like everyone technically has the choice to not own a phone in the first place. It's the "it doesn't affect you whatsoever" part I take issue with.