Those are separate issues: security experts would worry less about AV software hooking things if the AV developers were demonstrably following secure coding practices. Instead, they're generally shipping 90s-style C code and not following good practice like using sandboxing or other defense in depth measures, which is how you get things like a single exploit in a format decoder granting kernel access:

https://googleprojectzero.blogspot.com/2016/06/how-to-compro...

Contrast with the kind of exploit chains usually required on securely-designed systems like iOS or modern apps like Chrome where they need to chain multiple exploits together to escape from the sandbox before being able to attack something else. The AV industry has earned that bad reputation by generally being unwilling to spend the money needed to apply modern practice.