As a person who does not necessarily enjoy drinking (nor does so very regularly), I would say that being sober around people who are drunk is objectively less enjoyable than being drunk around drunk people.

In fact a graph of my enjoyment as a non-drinker (in my experience) would probably be inverse to a graph of the enjoyment of the people getting progressively more shit-faced.