Xi will never be safe from the party. The moment he runs out of steam, and presents a chance for power to be taken from him, someone, out of many millions of ranked party cadres, is just ought to take advantage of the situation.
Remember, both Kruszev, and Deng were relatively small fish at the time they seized power, as was the case for power takeovers in a lot of single party regimes.
Kruszev and Deng wouldn't have been able to seize power without substantial supporter.
The way I see it, the failure of those in power (from their perspective) was of not recognizing that well of support as it built.
Or, put another way, this is why authoritarian regimes are always paranoid: they have no objective methods for gauging public / party / military support, as dissent is always private.
(Well, aside from secret police)