I appreciate the take that the future harm isn’t worth the benefit today, since we’d enable future arbiters to control what is available.
There’s nuance in determining what is a public sphere vs. a privately controlled platform. The places with the most distribution are currently private, but crypto could change that to where we collectively own the platform, effectively making it publicly owned. Does that change your thoughts on when censorship is appropriate?
But I would not trust any corporation (sometimes it's profitable to remove something so they retain control of some market) or government (sometimes it secures their power to keep people unaware of some facts about their actions) to do the same. Would you?
So I guess the question is, how does the use of this new power to suppress abuses of speech weigh against other forms of abuse that become available to those who wield this power?
For clarity: this shouldn't apply to information that a corporation produces, because corporations aren't people and shouldn't have the same freedoms. Open source is still good, and controlling information that you discover, not that you invent, is unreasonable.
Overall, my issue is less with control over information (because if you believe all control is bad, then you have to say that revenge porn isn't too big an issue, and that's not a position I'm personally willing to take) and more capital's control over information because of all the cases when capital is misaligned with the interests of the people who need it. All of these cases feel like they define technology. Lots of the history of technology can be understood as large companies (bourgeoisie) trying to keep information hidden fighting a decreasing population of 'hackers' (the proletariat), and that fight's legacy is found here, on Google's hidden page gatekeeping what should be known.
...but that's not always the case. If your information has been copied against your will in a way that removes that key (which is incredibly hard to prevent, see the "analog hole" problem in video DRM), you have to have some sort of central authority or master key to be able to remove it, and then you're again falling right back into trusting that central authority with the power to remove whatever they deem fit. What if the central authority decides you don't own the information and not to remove it? What if they claim ownership of something they want removed that isn't theirs?
It just doesn't work. Information removal is a superpower I do not trust any large organization to wield responsibly.
The power to ban information is too great to be entrusted to any authority at all. Depending on how thorough the "ban" (web text filter at the ISP level? mandatory AR implants at birth filtering banned content? worse?), it's anywhere from an abhorrent violation of human rights and the principles behind free exchange and scientific inquiry all the way up through literally the most powerful weapon which could even theoretically be designed.
This is not a road worth going down, for any amount of harm reduction. The cost today may be worth it. The cost long term is potentially too great to even consider risking. There is no guarantee of who holds the ban hammer tomorrow.