At a minimum, Apple's review process includes a scan to ensure your app submission doesn't include private API usage.

Like private APIs that can screw with color output, outside of the app. And so on. The security comes from

1. Defining a set of APIs that you're allowed to use

2. Ensuring that you only use those APIs

Sideloading apps skips that important security check.

Step 2 doesn't happen. The private API scan is superficial and basically only catches accidental use (for example, maybe somebody discovered the API in a StackOverflow post and didn't realize it was private). It doesn't catch use that people deliberately want to get through. Getting through the check is as easy as some string obfuscation and a dynamic symbol lookup.

If a given private API is a security problem (and I agree that altering the screen's gamma settings system-wide may well qualify) then that private API needs to be actually secured, by taking it out of process and gating it on a sandbox entitlement.

The private API check is done to allow Apple the flexibility to change those APIs without potentially breaking a ton of apps and pissing off their users. It does nothing whatsoever for security, unless your threat model is programmers capable of writing malware but incapable of obfuscating function calls.