That's a decent argument to say immigration wasn't a real concern (that was the second counter point I mentioned). Although it does seem like a fair bit of Leavers voted because of migration (at least according to polls).