I don't think that's a fair characterization of Mastodon. But that's how many users see it, because that's how it has been advertised to them by media and other users. (And some instances have taken it as a slogan or a goal, e.g. apparently the authors)
If it's a background virtue it means it's not important, and if it's not important, then it won't be followed up when the pressure is put on. Some communities founded on principles of free speech (though not wholly operating that way as such) have turned out fine; certain boards on 8chan are ripe with intelligent and informing discussion, with a very strong undercurrent of free speech and against censorship.
The misconception here is that A) this is unique to 8chan, that those discussions are some how special and not being repeated across pretty much all online communities everywhere, and that B) this is the result of the lack of a dogmatic view of free speech that even evicts legal censorship of illegal activity.
This is exactly the same problem Voat had. Unfortunately it's pretty much inevitable whenever any kind of online space is explicitly founded as "[existing service], but without moderation!". Almost by definition, the people who are going to go there first, and form the basic norms of the community, are going to be the people who felt the impact of moderation on the old service-- i.e., trolls, immature brats, and people who want to post CP.
Freedom of speech is good, but it has to be a "background virtue": any online community explicitly founded on freedom of speech is going to be a garbage fire.