"Computer says no" is an incredible force multiplier for Authority, and anything that threatens the truthiness of the black box's output will be mobilized against.
Unaccountable security-through-obscurity is actually a bonus for Authority, too. Look at the whole Signal-Cellbrite [0] thing: Moxie found all these vulnerabilities, some going back years. What are the odds that an avatar of Authority('s technically-minded underling) found one or more of these Cellbrite exploits and used them as tools against enemies, or to exonerate friends? "Make sure you scan this cellphone first." Would any of that be possible if Cellbrite was legally mandated to be built from remote, publicly-viewable source every time they wanted to use it? Or, if you want to be less conspiratorial, how many vulnerabilities would Moxie have been able to find if the source had already been public (and subject to pull requests) for years?
The article just linked to the actual text of the bill but this summary seemed reasonable.
https://www.lawfareblog.com/artificial-intelligence-act-what...