There's a big difference between having voted to remain vs want to overturn the result of the referendum. Luckily most of those who actually were in the latter camp were kicked out by their electorates in 2019.