Xi will never be safe from the party. The moment he runs out of steam, and presents a chance for power to be taken from him, someone, out of many millions of ranked party cadres, is just ought to take advantage of the situation.
Remember, both Kruszev, and Deng were relatively small fish at the time they seized power, as was the case for power takeovers in a lot of single party regimes.
Kruszev and Deng wouldn't have been able to seize power without substantial supporter.
The way I see it, the failure of those in power (from their perspective) was of not recognizing that well of support as it built.
Or, put another way, this is why authoritarian regimes are always paranoid: they have no objective methods for gauging public / party / military support, as dissent is always private.
(Well, aside from secret police)
The timer on him is still ticking. He himself may well secured an n-th term, and party's temporary obedience at gunpoint..., but I feel he is very desperate, and have sleepless nights in anticipation of 2022.
Any fault of him, whether it is flagging economy, or any concession on the foreign front, or him not scoring just enough of political capital on that, would be 1000% used to undermine his position by whomever wanting to unseat him.
Unlike the oligarchates of ex-Union, CPC has very real internal conflicts. CPC's size ensures that there will never be enough posts, titles, and entitlements for every member of the establishment.