Just to clarify the phrase "one of the NTP people": I'm the lead designer of NTS, which adds modern cryptographic security to NTP. I did not design NTP; Dave Mills did that, in large part before I was born.
If I got the chance to redesign NTP from scratch, there are a lot things I'd change, but use of fixed binary fields is not one of them.
JSON is a disaster for many reasons. Hardware incompatible floating point is one; inconsistency in parser implementations (and ambiguities in the spec) also don’t help.
Also, why use a tree structured data representation when the underlying data structure is fundamentally just a N-tuple with a fixed schema?
Similarly, why use a text protocol to send around fixed length blobs or encrypted data?