Free movement enhances trade. The US doesn't restrict people from moving between states. The UK wouldn't accept Scotland trying to reduce people from other parts of the UK from working or living there.
> a negative to basically everyone
That's a politicised absolute which is clearly false. I think your political biases are colouring this.
I go back to my previous example. The UK doesn't try to impose government control on movement between its constituent countries, and the US doesn't limit movement between states. Freedom of movement improves economic outlook.
All EU countries are reasonably affluent by global standards. Some countries are richer than others, but it doesn't make economic sense to move from a country where you have a job to claim benefits in another one.
EU countries aren't obliged to pay benefits to people who have never worked in the country, so your idea that "governments will pay for benefits" is a very simplified view.
CMU is the EU trying to take control of the financial industry via their usual trick of "harmonizing" rules rather than allowing differentiation with mutual recognition. It doesn't address the problems actual voters had around unlimited immigration. That is taboo for the EU.