Glad you picked up on it. It’s not on the protocol guide yet but there are two or three new formats in discussion which all just sign opaque bytes. Wider adoption pending but the meta feeds is in JS already.

Here is the most recent one and it links to the other two as well: https://github.com/ssb-ngi-pointer/bendy-butt-spec

ps: I did implement the v8 pretty-printer in go and it was a nightmare.. I’m sure it still has some corner cases that are not covered….

> Don't sign abstract messages, sign bags of bytes.

Isn't the relevant question, "what are the bytes that are being signed?" That's what the specification seems to be answering here.

I believe it's implicit that the signature is actually of a hash of the message itself, which is why they need to serialize the message in a deterministic fashion, because otherwise the hash would be different in different circumstances for the same message. If you want to validate the signature, you need to make sure that the hash being signed is of the same exact string that you are verifying.

> requires canonical representations, a major PITA and source of bugs

Yes, absolutely, especially with regards to JSON, which doesn't have a fully deterministic serialization standard. The scuttlebutt specification seems to assume that JSON.stringify() will produce a consistent output, but that is not really the case, per my understanding, at least not with regards to objects.

From the Scuttlebutt specification:

> The canonical format is defined by the ECMA-262 6th Edition section JSON.stringify. For an example, see how the above message is formatted.

JSON object serialization in Javascript depends on the insertion order of the members of an object, so if you somehow change the ordering in which your keys are updated as the object is built, you will get a different string output, even if the data of the underlying object is the same. And this is just within the JavaScript runtime—if you are implementing this using another programming language, you can't just rely on the ECMA standard to determine the ordering of your object members at the time of serialization.

If I recall properly, JSON serialization is ambiguous in the specification in a variety of ways, which results in different implementations outputting subtly different strings, even under ordinary circumstances.

It seems weird that you would write a formal, generalized protocol specification relying on the idiosyncratic implementation details of JavaScript, for such an important thing as cryptographic signatures, as the Scuttlebutt specification seems to do here.

> It seems weird that you would write a formal, generalized protocol specification relying on the idiosyncratic implementation details of JavaScript, for such an important thing as cryptographic signatures, as the Scuttlebutt specification seems to do here.

The Protocol Guide was created after the initial implementation and its protocol were already in wide use, and the quirks were discovered while re-implementing it. More info about implementations here (Node.js, Go, Rust x2, and Python; additionally there are implementations of varying states in Java, C, Haskell, Erlang and probably others): https://dev.scuttlebutt.nz/#/?id=implementations

---

If making a new protocol using signatures over JSON objects, one might use this: JWS Clear Text JSON Signature Option (JWS/CT) https://datatracker.ietf.org/doc/html/draft-jordan-jws-ct JWS/CT uses JSON Web Signature (JWS) [RFC7515], JSON Canonicalizion Scheme (JCS) [RFC8785], and I-JSON [RFC7493] (subset of JSON for interoperability)

Or for signatures over JSON-LD objects / RDF datasets: https://w3c-ccg.github.io/ld-proofs/ https://json-ld.github.io/rdf-dataset-canonicalization/spec/

Matrix does it like this as well. Servers have to serialize an object as JSON (specified as Python code), then add the signature to the object and serialize it again to send it on the network. https://spec.matrix.org/v1.1/appendices/#signing-json

Exactly this. You need deterministic serialization, because you need to be sure that when the _same_ object is constructed in different settings, it is going to hash consistently. In Hyper Hyper Space [1], the set of basic types as well as the composition primitives used to construct all data structures have built-in deterministic serialization, just for this reason (e.g. a set will serialize into a deterministically ordered list, etc.)

[1] https://www.hyperhyperspace.org

> Exactly this. You need deterministic serialization, because you need to be sure that when the _same_ object is constructed in different settings, it is going to hash consistently.

I can see how this might matter in some specific systems, but when we're talking about signatures only the signer constructs the object. Stuff like the "JWS/CT using JWS and JSON Canonicalization" recommended in a sibling comment is a complete misdesign for virtually all signing use cases. That's why "our signature scheme requires canonical representations" is a red flag.

But "the signer" here is a cryptographic identity, that may be present in more than one device. So, even when conceptually it is just one entity, in practice it may be several computers doing something independently, and one may need the result to be the same given identical inputs.

Offhand that sounds really cursed: Signing the "same" thing multiple times with the same key shared among multiple devices and also expecting the signatures to be identical. I honestly can't think of anything that would require something like this and it sounds like a really bad idea as well.

Just to clarify something I've seen a few siblings confused about: Saying your signature scheme should not rely on canonical representations doesn't mean you shouldn't parse the input properly before processing it (e.g. checking that the input corresponds to a schema). It just means those two things are separate things best done separately. It's a little like NATting and firewalling: often confused with each other, or people arguing one acts like the other, but they're really separate things, and everything is much easier once you understand and heed that.

If you need determinism, I recommend avoiding objects and using pairs of arrays, plus they can generalize to matrices. Just use the same index in all.

On the contrary, lack of canonical format means plenty of room for implementations to fiddle with the format and introduce bugs and incompatibilities. The same bag of bytes ends up parsed as two different things on two different systems, which leads to SSRF-like vulnerabilities.

Throw away anything at the first byte that breaks canonical representation, don't bother verifying its signature.

> Throw away anything at the first byte that breaks canonical representation, don't bother verifying its signature.

IMO, the opposite order is better because it means you verify that the message is sent by someone you minimally trust before exposing parsers, etc, to attack, and someone attempting attacks must sign the messages. Here, getting an identity is so easy that it doesn't make much of a difference.

As far as attack surface, the cryptographic implementation is hopefully smaller than a higher level parser. The performance implications are unclear (both parsing and signature verification can be expensive).

If you do that, then there's no point in using JSON -- you won't be able to use preexisting JSON parsers or writers anyway.

Meanwhile, the sign-the-bytes crowd can use any preexisting format.