Let's suppose, for instance, that Twitter is deemed large enough to be a "public forum" and no longer makes moderation decisions outside of removing illegal content. The clear, obvious consequence for such a decision would be that most people would cease to use Twitter. It would no longer be a "public space" because that "public" would no longer be there.
People would stop using Twitter for the same reason the public doesn't use 4chan. Anecdotally, I don't want to be harassed for my sexuality on Twitter. I wouldn't feel safe, or want to participate in a site that allows open attacks against people due to their gender identity, race, or religion. And let's not kid ourselves, the "conservative" view points being "censored" on Twitter aren't really "conservative" views at all, it's just hate speech, harassment, and attacks against marginalized people. Even semi-famous self-described fascist content creators continue to use Twitter above radar, provided they don't explicitly distribute hate speech on the platform.
Marginalized people of every form would find another place that is moderated to flock to. That would become the new "public place" that so-called "conservatives" would wish to invade.
The ability to control the "public" spaces gives one effective control over "speech". That's a lot of power.
Having that power entirely outside of democratic control troubles me.