This has nothing to do with logic. It's a knee-jerk reaction, and the pro-Brexit propaganda played it hard (alongside other similar arguments).