I'm unfamiliar with kernel development cycles, but there might be some amount of maintenance needed each patch to ensure changes work for the various supported architectures, in which case leaving them in without updating them would result in insecure, increasingly buggy mess.