The arguments I've come across fall into three buckets: neighborhood character, traffic/infrastructure, and (my personal favorite) supply and demand doesn't apply to housing. I suspect the real root is a fear of change mixed with greed.