Golf, generally, is pretty expensive. It's like minimum $50 for an outing, you need equipment, correct clothes, etc. Some places require membership, often priced intentionally exclusively. It's pretty natural for something exclusionary to get a negative cultural bias.

Oh, and it is a terrible resource hog. You can't fit many people on a golf course at any given time without disrupting gameplay, and all that grass requires a lot of water and maintenance.

> any reason to spend a couple hours outside with my friends sounds amazing

This is, of course, available in many forms that don't involve hitting balls with sticks, but also there are many varieties of ball+stick that satisfy this.

Golfing is an artificial competitive activity that exists in an artificial and manicured version of nature. There is nothing wrong with it if you like the activity, but you can just go for a hike or stroll in a park if you want to be with friends outside.