Edit: Even if the RPT wrote their own firmware, not from Broadcom, and released it, they would still risk being sued to pieces for patent violations galore. After all, if you license H.265 alone, there are over 1,100+ pages in the Table of Contents alone just listing all of the patents worldwide on it.
Edit 2: And they could be possibly sued by Broadcom for violating the IP sharing agreement used for developing the chip by developing an open-source blob software.
If I buy your patented thing from you, that exhausts your patent rights for that copy of the thing. If I modify it, and do something you don’t like with it, you can’t come after me under patent law.
Note that I didn’t say anything about contract or copyright law—they may have signed something saying they wouldn’t reverse engineer the chip, or the sale could come with an end user Eula to that effect.
I'm just saying that if RPT tried to write their own firmware, they could be in big legal trouble, so I wouldn't expect any open-source firmware from RPT anytime soon.
I think you're confusing copyright and patents. Just because some technology is patented, it doesn't mean it can't be open sourced.