That is how he presents it.
There doesn't have to be a conscious, collaborative effort by Your Politicians and Their Politicians to keep the Voters screaming at each other, so that they don't notice the increasing gap between the Voters and the Politicians. There doesn't have to be a conspiracy. It emerges from the interests of the individual politicians in getting you to identify with them instead of judging them.
He calls it a "swindle" because whatever reservations politicians have with their party, they all try to get you to identify with themselves.
I'm sure a huge number of politicians are very frustrated by many of the policies of their party and would like to diverge from them to a greater or lesser extent. They are constrained too.
Why can we not see this as an emergent phenomenon of a complex system, with little overall direction?