No it is not outside the realm of possibility that people could conspire to perform a double spending attack, you are completely right. What you have to consider, however, it is not very likely that people conspire to undermine their own economic interest. That doesn't happen very often in this world, it is not in our nature (but it is a possibility).
There are much cheaper and easier attacks that could be pulled off right now by a single attacker, lower hanging fruit (like flooding the network with transactions with above average fees, this will effectively block most other transactions for as long as the attacker can afford).
If a double spending attack was to occur at the mining pool level, you wouldn't even need to have a conspiracy to make it possible, just a single large pool operator with a period of extraordinary luck would be able to get 6 blocks in a row (it has happened before, it will happen again).
If pool operators conspire to double spend or change the protocol there will be a mass exodus of individual miners from the pools.
Bitcoiners are savvy enough to get raised eyebrows if a major pool started encouraging and/or requiring users to use a proprietary, closed mining software that may attempt to modify the bitcoin network's behavior to do something unsavory.
Yeah, this sounds like exactly the sort of thing that people would do.
One would think that the leading figures of these pools know each other. One would think it's outside the realm of conspiracy to think that the designers of Bitcoin expected the mining pools to emerge once difficulty had grown sufficiently large.