It can work out O.K. if you stick to patching srpms of the official releases. For example, in the past I've needed to patch performance fixes for RAID cards into the kernel. When set up carefully, this is easy, repeatable, and has minimal impact.