The downside in public is that the people who notice you're speaking another language will assume that you're talking about them and don't want to be overheard by the victim.

It is less of a big deal than when you're in a social group, where someone could feel that you are deliberately excluding them.