There were issues with France due to them leaving and the way they chose to, but the issues with the Irish border were constraining the way they could leave (to some level)

The UK could choose to leave and cut off all agreements concerning fishing rights, the tunnel, the border checks with france and it'd have been perfectly legal, if diplomatically damaging

With the Irish border there was/is the complication of the GFA, which prevented the same kind of approach