Which is strange because JavaScript has had the Uint8Array that makes manipulating arbitrary byte structures easy for a while, it doesn't feel any less natural than other dynamic languages in that regard. Defining a "signed message with a header" structure in terms of bytes like I described above seems like a no-brainer. If you want the payload data of your protocol to use JSON then that's fine, but that shouldn't dictate the cryptographic layers underneath.
https://bnfc.digitalgrammars.com/
I agree, having/using a well-defined standard that can have automated tooling to general language-specific implementations would be ideal.
Indeed, this has already created considerable problems for the request and event signing and getting different Matrix implementations written in different languages to interoperate correctly. I now firmly believe that cryptographically-signed JSON is always a disaster waiting to happen, no matter how hard you try to canonicalise it.
We've done our best to make sure that the way we're serializing objects can either be done in a consistent manner across implementations or, if done inconsistently, it doesn't matter.
The consistency is really an important part here, are you spending less time doing the testing and bugfixing to ensure that JSON ambiguity is not possible / is harmless than you would writing serialization for cryptographic types in unambiguous byte-oriented structures?
Nailing all that down so you can e.g. have a truly deterministic signing process often requires custom encoders/decoders and is rather fraught with problems even then, at which point why even bother with JSON. Use something designed to be consistent.
But a healthy ecosystem of clients never emerged. When I looked into why, I discovered it relied on node's particular method of JSON serialization[0]. Are you confident that the shelter protocol has avoided this pitfall?
However, thinking on it more, there is one thing Groxx mentioned that has given me pause, and that is "Numbers don't have a specified precision". This could be an issue in the future if, for example, computers suddenly use 128-bit numbers and JSON implementations start to actually use such numbers. It's an edge case that might result in incompatibility with clients that use 64-bit numbers. So we'll consider the possibility of changing the serialization format.
I asked Brendan Eich if JavaScript will ever change the way it represents floating point numbers (from 64-bit IEEE 754 values), and he said no ("don't break the Web"). And to represent larger bit values in JSON you're supposed to use a JSON string and schema to specify the type. I'm not sure exactly what he means by that, but I assume it means something like using JSON keys like "<BigInt>mykey" and string values instead of number values. Then manually parsing the string into whatever native 128-bit type or whatever representation you have.
So - JSON still seems to be the favorite here, but I totally get the anxiety around it because of how under-specified it is. If you avoid re-serialization nothing should break.
Like the SPMessage format here too, uses JSON.stringify to indicate structure encoding, which necessarily implies that any implementation of the protocol is bringing along needing to handle JSON semantics. JSON does not seem to be well-suited here as it doesn't composing cleanly in the way a data structure defined in terms of bytes would.