> If the other party had the same information they would have sold or bought at a different price point. (or maybe not at all+) That has to be because that's the attraction of insider trading

This doesn't follow at all. Trades happen all the time between people with the same information. Under your theory, that's impossible.

It's also not particularly relevant to the main point. The idea of banning insider trading isn't to reflect inside knowledge in the price instantly -- that would be accomplished by encouraging insider trading. The idea is to prevent people with inside knowledge from trading. People who sell to someone trading on inside knowledge get a little bit more money (due to higher demand) than they otherwise would have. They miss out on gains that the person with inside knowledge predicted, but that they also miss in the relevant counterfactual (when they sell to someone else with no inside knowledge).

A few end up selling when, in the world without the insider, they wouldn't have. Those people can't be identified and quite plausibly realize the same gain under either scenario, for example, if they have a standing order to sell at X price.