The app makes sure only the people it thinks are in the channel get the keys. If you turn on the security notifications you can see how it works, you get a message when someone gets a new phone, that indicates they got a new key.

You can even check if it matches up if you meet in person or if you communicated using another medium.