From living abroad in a few different countries, this just isn't the case for most people unless a specific effort has to be sought out to find similar people. A large number of people will simply default to hanging out with people who speak the same language as them when they want to relax. Most people, but obviously not everyone, until they reach a certain level feel slightly uneasy in a situation where they are the only non-native. I've been in the situation many times and had a good time, but the anxiety is there. What I found the most interesting was that non-native but well spoken English speakers were also more comfortable with English speakers while they were learning a new language.

When you aren't forced into the situation, say living in the countryside of a foreign country, or you aren't extremely studious its fairly natural to spend your down time with native speakers.