Converting a text format to a compact in-memory data structure takes extra CPU cycles. (de)compression takes extra cycles. The point of using a binary format is to achieve the same result while avoiding that overhead. For some data formats compression also has the downside that it prevents seeking.

JSON doesn't allow any custom type, so it is not "flexible" per se. Therefore you only need a format that supports the JSON data model and pretty much nothing else; CBOR [1] for example almost surely fits the bill.

[1] https://cbor.io/

Serialized binary encodings don't require a schema. They can have all the expressability of JSON and more.

That’s an interesting thought. I am now very fascinated by what kind of data would be gigs in size but would need the flexibility of JSON.

Datasets are not always provided by you; if your data source outputs json it doesn't matter why. Also just because your volume of data is measured in gigabytes, that doesn't mean it's a singular stream. As an example you could be handling tens of thousands of small requests.