Whether the kernel can vbe audited is completely orthogonal to verifying it at boot time since there are still ways to retrieve it even with active hardware encryption.

The utility of patching your own kernel has to be carefully weighted against the security impact of doing so. If an attacker can take over the user account that compiles the kernel, they can make you install a compromised kernel. For example, by running a file system monitor that exchanges the kernel image with a compromised one as soon as it has been built.