Thanks for the response! These are very good references. I like the breakdown of message and/or service definitions into separate files so it is easy to browse through the repository. I am assuming the version bump (v1 vs. v2) occurs when you introducing breaking changes to the API? I am aware that protobuf does a good job at allowing forwards and backwards compatibility at the message level..
Correct on the versioning. We internally treat protos as an immutable descriptions of our API interfaces. That means whenever we need to change anything (outside of bugs), be it adding a new field, or changing the order, renaming fields, changing types, ... we start a new version.
We also use a lot of inheritance of non-default types (timestamps, errors, ...) so it's important to make sure that we don't break anything for others.
Interesting so adding new fields even.. I suppose there is quite a bit of planning and iterating on the interface before making it public to reduce churn? I understand doing that for backwards incompatible changes.
We add new fields (or redesign existing protos) very sporadically. We also don't have fully automated process to deal with a change in a proto within all projects so by using new versions we can signal to the developers that they should eventually adopt it. We are actively using 5 languages that have to deal with the changes, so we found it easier overall to do it this way.
Re:gRPC-Go Engineering Practices
by marshallbrekka on Friday January 26, 2018 @01:55AM
(#16237475)
The mono-repo of protos has been our approach, and has worked very well for us so far.
Only difference is we publish the resulting code for all languages into a single artifact repo, which other projects use as a dependency.
First challenge is, that you have to keep some kind of reference on what proto are you using within your project. What Google (and some others) do, is that they a) put all the proto files in a separate repo [0] and then generate them for each language separately (python [1], ...). This way you can use whichever proto file you need within your project, however you have to load more libraries than you need to. To be honest, it only makes slight difference when deploying, so not too bad.
The second challenge is, that you have to generate the result files every time you make some kind of change. If you have a lot of proto files, then it may take some time to generate them and there are very little tools available to help you. Google open-sourced Artman [2] although it's more focused on APIs than managing shared protos.
The massive advantage is that, because proto files are self-explanatory and if you put enough information in them can function as direct documentation of your API's interface, you don't need to fish out the requirements in the project or in the documentation but rather just directly read the proto file itself. But this does depend on the developers, to make it as consistent as possible, which is not always the case [3].
[0] https://github.com/googleapis/googleapis
[1] https://pypi.python.org/pypi/googleapis-common-protos
[2] https://github.com/googleapis/artman
[3] https://news.ycombinator.com/item?id=16166153