I only ever use a half dozen or so public Matrix rooms (I mostly use it for private chat) but my experience in those is nothing like you describe. Unsavory characters occasionally wander in, just as they do on every other public forum I've used, but they are shown to the door and the normal conversation carries on without them.

I wonder how someone must be using the platform, or how they're looking for rooms that might interest them, to get the impression that it's "mostly pedos and far right extremists or other weird anti social characters."