I would not necessarily leave, typically in a situation where theres an annoying person present, me and my friends would try to get rid of the annoying person, and if that keeps failing, leave for home...