I think it's fine if you don't re-encode (i.e. need to decode) the audio/video packets in any way. IIRC, you can broadcast RTP stream packets without knowing their contents if you settle on common codecs (e.g. opus and h264). But if you need to embed picture-loss-indicator messages or something, it might not work. I think this is the essence of "SFUs" (selective forwarding units) in WebRTC parlance.