jhumphries131

I agree! Protoc allows comments anywhere, but it doesn't bother preserving them all in the descriptor. At one point I thought this was a bug, since comments _could_ be preserved in far greater contexts (though definitely not all). But then I realized that the descriptor.proto comments do actually state the places where comments are retained:

> If this SourceCodeInfo represents a complete declaration, these are any > comments appearing before and after the declaration which appear to be > attached to the declaration.

https://github.com/protocolbuffers/protobuf/blob/v21.5/src/g...

After re-reading with this caveat in mind -- that comments are only preserved before and after complete declarations -- I realized that it does retain the comments that are expected.

Long story short: the descriptor is bad as an AST if you care about recovering the original source. The descriptor is lossy. I'd recommend using a real AST for use cases that want something non-lossy: https://pkg.go.dev/github.com/jhump/protoreflect/desc/protop...

Gob is Go-specific. Our mission is to make schema-driven APIs easy, regardless of what language you use. Protobuf already has official support for nearly a dozen languages, and unofficial support for many more. Protobuf also has a compiler with a plugin model, which facilitates supporting even more in the future.

Furthermore, Protobuf is an IDL, not a full-blown programming language. This makes it ideal for this use case, for describing APIs and data structures.

Gob-encoded data structures are described with Go. While Go is great for writing server-side business logic, it is not as well-suited as a description language for data that you need to share with non-Go systems.

Our aim is to make the spec accurately match Google's reference compiler -- for as long as that is the source of truth, which is hopefully not forever :)

Even for those not using Buf, we expect this documentation to be of interest to the community as it describes a large number of facets of protoc that were previously undocumented (and required examining the source for protoc or playing around with test source code to see what it expects and what descriptors it generates).

If issues are found with this spec, it is true that we'll most likely have to revise the spec to match the compiler, not the other way around. But no software is perfect: some variations will be due to bugs in protoc, which can be fixed in the compiler to properly match the spec. Over time, we'd love to see an outcome where a formal specification is the source of truth.

For now, our commitment is to make (and keep) this spec as accurate as possible to describe the Protobuf language, not some Buf dialect.

It's possible that the internal version of protoc is very different from the open-source version. (I know there are numerous differences, but not sure how pervasive they are in the parser.)

The open-source version has a hand-written tokenizer and recursive descent parser that is not too difficult to translate to EBNF. You'll notice that the section on numeric literals is a little wonky, because the tokenizer does a check that is hard to describe in EBNF. But it isn't too bad.

Also, some of the constraints of the language are in prose in this spec because they are easier to enforce using a semantic validation pass, instead of trying to model purely with a CFG. (Optionality of the colon in the text format, used in message literals, comes to mind.)

There are some things that technically _could_ be handled in the grammar, but they would make the grammar much more cumbersome to read and understand. So those things are also extracted into prose.

> Definitely interesting for this company to create an EBNF definition for protobuf.

For what it's worth, Google has also published an EBNF definition (the subject blog post contains links to those specs). But they are incomplete and not entirely accurate, which is a non-trivial part of what led us to writing and publishing this spec.

The key "killer" use case is for describing RPC schemas. By describing the schema in an IDL, you can then generate client stubs and server interfaces in a variety of implementation languages, allowing interop between heterogenous clients and servers.

They are also useful for describing domain models. This isn't surprising since domain models usually find their way into RPC schemas (since RPCs will often query or define model data). But they can also be used in other cases, such as for persistence and structured logging.

Some misuse I have seen involves trying to make a protobuf model the _only_ representation of a domain model: it is almost inevitable that a physical model (a representation of your data in a SQL database, for example) will need to vary from a logical model, and trying to make a single protobuf representation serve double duty can be a source of problems. Making protobuf schemas conform to physical storage constraints often makes for worse abstraction. The model becomes de-normalized (which can make constraints and relationships harder to model/enforce), and can even leak details that consumers shouldn't know about or care about.

Another misuse is using it for data that never leaves a process -- a program's private, internal state. If a data structure never needs to be serialized (to persist or send to another process), then you're better off using native data structures in the implementation language (which generally have far greater flexibility/expressibility as well as better performance).

It is a simple language, being just an IDL (no expressions, no logic, no state, no memory model, etc).

However, you might be surprised about the room for ambiguity. For example, there are several mistakes in the grammars on the official developer site. And there is no place on the developer site that, for example, clearly explains how option names are formulated and interpreted. Even the way that relative references are resolved is not thoroughly described; it is probably the most complicated part of the spec because coherence/consistency wasn't keenly considered when the reference implementation in protoc was devised (https://www.protobuf.com/docs/language-spec#relative-referen...).

So if you wanted to write a tool for the language, one that could correctly parse and understand a source file, without the details in this spec that tool will almost certainly be incorrect and disagree with how protoc parses and understands the same source.

Our intent with the compiler in Buf is to match protoc as perfectly as we can. We want to instill maximum confidence in our users that Buf is a trustworthy tool and a suitable replacement for protoc. And to do that, we need the behavior to match.

But we do hope that eventually the official definition of the language will be a proper specification, not a particular implementation. (And maybe this document could be the start of that shift.)

So while there are multiple implementations (Buf, Square/Wire, probably others), the protoc implementation is canon.

We keep an eye on the protobuf repo, so if a change is made we can both incorporate it into our products (https://buf.build) and into this spec.

A great outcome for the ecosystem would be that Google chooses to engage with the community before making language changes. And the best possible outcome would be that the authority is eventually inverted: a specification doc becomes the definitive source of truth on the language and `protoc` is updated to conform to it, instead of vice versa.

A great question: We do plan to add content about the plugin protocol to this site. While documentation for plugins is light, it is easier to find and to get a working plugin than it is to find the information needed, for example, to write a tool that performs static analysis on a protobuf source files.

The biggest omission in the existing docs was the specification of the language.

Plugins generally require a library for a particular implementation language, so content we write would likely have to focus on a single language and library (at least to start). Whereas a spec is more broadly useful, regardless of what implementation language one is using with Protobuf.

As of right now, the former. That is currently required for our tools to remain functioning (https://buf.build). If/when changes are made to the language, we update our tools (and this spec) to continue to be accurate.

Google actually requested that the community contribute to a real specification. https://github.com/protocolbuffers/protobuf/issues/6188#issu...

So we've taken the initiative. No rent-seeking.

The intent of this spec is to actually put "whatever Google did in protoc" into a readable format, so you don't have to read the C++ code. The official docs fall short on providing much of the details that are included.

Matt, that is not our intention. But we are trying to build a business around making schema-driven APIs easy, and protobuf is at the core of our current products. So we are trying to improve the ecosystem around protobuf, and a critical aspect of that in our esteem is having a spec.

While `protoc` remains the source of truth, this spec captures the syntax and rules accepted and enforced by `protoc` in a far more detailed way than the official developer site.