Key difference is, the founders and leaders of Judaism and Islam claim to have believed in God. The Buddha did not. A lot of us identify a deity as being core to the idea of being a religion.
That's not to say that sects haven't developed and diverged from the initial beliefs, perhaps in order to build their own communities (or empires). But these are later developments than the teachings set forth by Siddhartha Gautama.
if you read even just a bit of 2000 years history of Christianity, you'll see that violence in the name of God and Church and toward the infidels/heretics/etc. has all this time been considered a virtue of a good Christian. Not that Christianity is any special here. The base of any religion is division between "good" and "bad" and fostering the feeling of superiority in "good" and hatred toward the "bad". In particular "<Religion> of peace" for any value of <Religion> is an oxymoron the same way as "white black".
Wikipedia has a light page on the topic.
https://en.m.wikipedia.org/wiki/Buddhism_and_violence
Not mentioned in the above link is the Tibetan serfdom controversy.
https://en.m.wikipedia.org/wiki/Serfdom_in_Tibet_controversy
Regarding you second point, I am not sure I see the value of saying "Buddhism is not a religion if you ignore the religious bits"... you might well say that about any religion. There are plenty of secular Muslims and secular Jews for example, but that does not make Islam or Judaism any less of a religion.