Note that if you can compile the kernel yourself there are benefits to doing so. It's been a while so I forget the details but grsecurity uses some randomization at compile time, and an attacker with access to the public image would be able to get around those defenses. If you own the seed, and delete it after compilation, an attacker will not be able to bypass those defenses.

Naturally you still benefit a ton without those, but the ideal situation involves compiling it.