News for hackers. Stuff that matters.
Re:Don't Use Protobuf for Telemetry by tijsvd on Wednesday December 30, 2020 @04:57PM (#25586639)
What is this thing with JSON support? Don't people use pb so they do not have to deal with JSON? I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size?
Re:Don't Use Protobuf for Telemetry by kentonv on Wednesday December 30, 2020 @05:19PM (#25586867)
People implementing public-facing APIs might not be comfortable forcing all the consumers of that API to use Protobuf, but may want to use Protobuf internally within their service. So, Protobuf lets you convert to/from JSON, so that your service can expose a JSON API while maintaining Protobuf schemas and never dealing with a JSON parser directly.

The implementation is based on Protobuf's existing reflection features, so if you are already compiling with that enabled, the JSON implementation does not add much code bloat on top of that.

Re:Don't Use Protobuf for Telemetry by joshuamorton on Wednesday December 30, 2020 @05:18PM (#25586862)
[Also a googler]:

PB is used for a wide variety of things, its a lingua-franca of data interchange within Google, but web (frontend) still often uses json. So if I want to have some proto in a database and render it on a UI to a user, somewhere in there I'm probably going to be translating the raw proto to either json or json-like (I'll admit I'm not sure how stuff like grpc-web fits in here, if you can just get proto-serialized bytes passed around in some a blob, but there are some pretty low-overhead json proto encodings that exist anyway).

Re:Don't Use Protobuf for Telemetry by tijsvd on Wednesday December 30, 2020 @06:04PM (#25587357)
Sure that makes some sense.

But it's somewhat ugly as well, from an architecture point of view. Because that argument translates to anything else you might want to do with these objects.

In past libraries for C++ I've tried to prevent this kind of coupling by adding generated template methods like "walk(f)" where f would be a templated callable, called with a descriptor and data reference for each field. Any kind of pretty printer or SQL statement can be built that way.

Re:Don't Use Protobuf for Telemetry by haberman on Wednesday December 30, 2020 @05:27PM (#25586955)
> Don't people use pb so they do not have to deal with JSON?

I think that's one reason, but there are others too, like getting generated type-safe accessors and an explicit schema. I've sometimes used protobuf for this reason even when I'm not planning to use binary format.

> I'd expect that for a truly lean pb implementation, adding JSON is a 300% increase in code size?

Here is a breakdown of a binary that uses upb and links in binary format, JSON format, text format, and some generated protos:

         VM SIZE
     ---------------
      20.6%  26.5Ki    Generated Reflection
        45.7%  12.1Ki    bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto3.upbdefs.c
        39.3%  10.4Ki    bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto2.upbdefs.c
         5.0%  1.33Ki    bazel-out/k8-opt/bin/external/com_google_protobuf/conformance/conformance.upbdefs.c
         3.1%     832    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/struct.upbdefs.c
         2.4%     648    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/wrappers.upbdefs.c
         1.2%     320    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/timestamp.upbdefs.c
         1.2%     312    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/duration.upbdefs.c
         1.1%     304    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/field_mask.upbdefs.c
         1.1%     296    [1 Others]
      19.1%  24.5Ki    JSON
        60.4%  14.8Ki    upb/json_decode.c
        39.6%  9.67Ki    upb/json_encode.c
      14.7%  18.9Ki    Reflection Runtime
        81.9%  15.4Ki    upb/def.c
        18.1%  3.41Ki    upb/reflection.c
      13.9%  17.8Ki    Generated Code
        46.9%  8.34Ki    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/wrappers.upb.c
        17.6%  3.12Ki    bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto3.upb.c
        16.4%  2.92Ki    bazel-out/k8-opt/bin/external/com_google_protobuf/src/google/protobuf/test_messages_proto2.upb.c
        14.7%  2.61Ki    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/descriptor.upb.c
         2.0%     356    bazel-out/k8-opt/bin/external/com_google_protobuf/conformance/conformance.upb.c
         1.4%     264    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/struct.upb.c
         0.5%      88    [2 Others]
         0.3%      48    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/any.upb.c
         0.3%      48    bazel-out/k8-opt/bin/external/com_google_protobuf/google/protobuf/duration.upb.c
      10.0%  12.9Ki    Binary Format
        54.0%  6.95Ki    upb/decode.c
        46.0%  5.92Ki    upb/encode.c
       8.8%  11.3Ki    Core Runtime
        66.3%  7.52Ki    upb/table.c
        17.8%  2.02Ki    upb/upb.c
        15.9%  1.80Ki    upb/msg.c
       5.6%  7.19Ki    [34 Others]
       3.8%  4.94Ki    Text Format
       100.0%  4.94Ki    upb/text_encode.c
       3.5%  4.46Ki    [section .rodata]
     100.0%   128Ki    TOTAL
So JSON is pulling in 25Ki directly, plus it depends on the 19Ki reflection runtime and the generated reflection.

But it is all a la carte: if you don't want JSON, just don't use it and don't link it in.

(btw that 7Ki for upb/table.c is to support protobuf maps, I want to make that a la carte also so that you don't pay for it when you don't use maps).

Re:Don't Use Protobuf for Telemetry by 1f60c on Wednesday December 30, 2020 @10:51PM (#25589403)
How'd you generate that listing? It seems useful.
Re:Don't Use Protobuf for Telemetry by haberman on Thursday December 31, 2020 @12:20AM (#25589853)
I used my tool Bloaty, which parses the ELF and DWARF information in the binary: https://github.com/google/bloaty
Re:Don't Use Protobuf for Telemetry by 1f60c on Thursday December 31, 2020 @10:03PM (#25600420)
Thanks!