While there was no obligation to grant permanent residency, the criteria that would allow removal before the five years (or two years in some cases) are narrow. If someone had a job and in some cases when they didn’t, it was impossible to remove them unless they posed a serious threat. That is, in fact, a significant limitation on the control the UK could exert over immigration from the EU.