Sorry for the somewhat sloppy phrasing. GUSD is 1:1 backed by USD, the bank accounts holding that USD are FDIC insured, and this arrangement has regulatory oversight: https://www.gemini.com/cryptopedia/gusd-stablecoin-gemini-do...

You're right that if you store your GUSD off Gemini and it gets stolen, then you're out of luck. I am not sure what happens if Gemini closes but my impression is that since they are a NYS trust they would have to their customers the 1:1 backing (rather than paying creditors with that money).