You're being naive. We mass produce animals for slaughter.

Those ants are pests, I pour boiling water down their holes and leave poison around my house because once in a while a few get on my counter, or they cover the sidewalk when they go to war with each other. I have a pet rabbit, vermin, but I put out mouse traps.

An alien species entire reason for making us extinct or pushing us back to the dark ages might be as petty as because we were a visual nuisance, or like Douglas Adams suggested: we got in the way of road construction.