Frankly this "they do deserve it because it keeps them happier" reeks of sheer entitlement and disdain for personal property and boundaries. I have had terrible experiences in the past with neighbour cats. And if you don't think cats are aggressive you clearly haven't seen a caterwauling fight mess up the neighbourhood or scratch kids up badly.

Also, many people are allergic to cats. Some of them develop even serious health problems like anaphylaxis. Please restrict your cat strictly to your property.