While there was no obligation to grant permanent residency, the criteria that would allow removal before the five years (or two years in some cases) are narrow. If someone had a job and in some cases when they didn’t, it was impossible to remove them unless they posed a serious threat. That is, in fact, a significant limitation on the control the UK could exert over immigration from the EU.

Well, yes, there was freedom of movement. I'm not sure why being unable to remove someone who has a job is so terrible. We could have chosen to put a cap on migration from relatively poor EU countries, but we chose not to.