Its normally partipants with bad mics, headphones, AGC and the crappy audio support in windows.
I do a number of actual play games and it's the players trying to use the built in mics/headphones that have difficulty hearing the group conversations.
I use a high quality external sound card and don't have that problem - hangouts does seem to disconnect me when I don't speak for a while as my noise floor is so low
It's a lot to process. With 15 people on "speakerphone", the audio is IMO the biggest and most taxing problem.