That seems a bit reductive. From my understanding, it's not so much by the wrong people but rather without deliberate, conscious choice. Chiefly, anabaptists reject the notion of infant baptisms because an infant can't meaningfully repent and freely proclaim their belief in Christ. Which, as far as religion goes, seems like a reasonable position.
I didn’t know this is what set anabaptists aside.
A lot of distinctive Anabaptist beliefs have gotten mainstreamed over the centuries, and what they were once best known for is now much more widely practiced. But the name sticks for historical reasons and the group still exists. You can read a summary of beliefs here: https://en.wikipedia.org/wiki/Schleitheim_Confession In the modern mileu, I would say what is most unusual and distinctive about them is pacifism.
Makes total sense; I don't know why you'd question it!