Don't dismiss him so lightly. There's a huge difference between infrequent bullying and sustained bullying. As you say, sustained bullying is abuse (in its modern sense), though we often use the words interchangeably.

Imagine that you're in a situation where you're thrown together with people who if they even see you will try to do something nasty to you. Now imagine that happening every day, for years.

I can see how it would change someone.