I don't understand a lot of high-level physics or Continental Philosophy, but the same process that leads me to believe that the physicists aren't sitting around BSing all day leads me to believe that the philosophers are.
(Which is to say, I read enough intro surface level articles that are supposed to give a hint of what you get into once you do begin to understand and talk to people about the same, and form an outsider's opinion. Same process, different results with the two fields.)
I'm guessing you aren't familiar with string theory then. :-P
/duck
PS: Guess which side won?
Sokal punked a minor journal that was not particularly well known in the field. His book with Bricmont, attacking "postmodernism", on the other hand, was published by a major publisher, even though large swaths of it had been thoroughly debunked.
I don't understand a lot of high-level physics, but that doesn't lead me to imagine that physicists just sit around bullshitting each other all day.