The manufacturer can set the invoice price at whatever they want already. Dealerships are only negotiating their piece of the pie, they aren't actually able to alter the core price of the vehicle. The market/competition determines what a manufacturer charges, the dealerships just obfuscate the exact figure.

So I agree with the other commentor, they exist to extract local money, to make a few locals rich, the manufacturer's piece of the pie is whatever they think they can charge either way. If you cut out dealerships you make prices less obscure and force manufacturers to compete more closely with one another, and also save the middleman rent-seeking.