In the YouTube video linked elsewhere in this thread at https://news.ycombinator.com/item?id=40805461, someone breathing in a balloon-full of xenon gas clearly struggles to expel it afterward, remarking that he can feel its heaviness, but also visibly succeeds after just a few deep breaths. I guess this could still be dangerous if you were fully unconscious and breathing shallowly (and absolutely if you were unconscious and wearing a device that kept delivering more xenon over time), but it doesn't look like it's literally necessary to turn upside down to get it out of the lungs.