Sure but was society better off because those were patented as opposed to open (or secret)?

Yes, and look at the history leading up to this(secret algorithms with a trust me it works non-guarantee). The cat was out of the bag on pki and there was no putting it back. People built on that knowledge and developed new techniques

Yes, because there's a reasonable chance they would not have been invented (or they would have taken much longer to appear) without the economic upside of a patent. Also, a patent trades forced disclosure for a limited monopoly. The disclosure in both of these cases has been very good.

Also, people immediately licensed the patents and it further research in the field that was owned by entities like the NSA prior

Very dubious, given that both have since been superseded by better patent-free alternatives.

Which are all built on top of the public disclosure of these algorithms, which was required by... the patents!

Prior to RSA being patented, a lot of encryption was proprietary (and probably mostly broken), and nobody could build on top of it. Post-patent, the RSA inventors could publish details and publicize their invention, which led to the alternatives.