Every feature, even the one you don't use, comes with a price. There's no such thing as free feature in software.

Adding sideloading reduces security of a device even if you personally never touch it. There'll be more attack vectors, scams etc that would only be possible with sideloading.