I read through your upb code about 3-4 months ago, was initially impressed, but couldn't get the Python extension to work. Certain abstractions really lost me, like pushing and pulling between sources and sinks. Why not just let a top-level event loop run the show in terms of buffered reads and size calculation for writes? But maybe you've refactored since.
Sorry, I should be clearer about the current state of the code, which for the Python extension is: currently completely broken. Since I was focusing on the core interfaces, the more peripheral pieces (like the language extensions) are totally broken at the moment.
> Certain abstractions really lost me, like pushing and pulling between sources and sinks.
Hopefully more documentation will make this clear. Making sources and sinks a general abstraction makes the event-based interface independent of any specific serialization (like protobuf binary format, protobuf text format, a JSON serialization of the same schema, etc). The key thing about protobufs is that a .proto file defines a typed tree structure, and the core interfaces of upb let you iterate over that tree structure, regardless of how exactly that tree structure was serialized.
> Why not just let a top-level event loop run the show in terms of buffered reads and size calculation for writes?
In my most recent interface, the upb_src does indeed run an event loop, and calls callbacks for every input value, or on a submessage start, or on a submessage end. For a while I wanted to make upb_src a pull-based interface instead, to give the application more control over the main loop, but this created more problems than it solved.
Suppose that the generated C++ takes 1.0 seconds. Does your implementation take 0.7-0.9s or 1.7-1.9s or something else?
https://github.com/haberman/upb/wiki
(if anyone's looking at the code, I'm working on the src-refactoring branch at the moment)
The benefits of my approach are:
* you can avoid depending on a 1MB C++ library. upb is more like 30k compiled.
* you can avoid doing any code generation. instead you just load the .proto schema at runtime, so you don't have to get a C++ compiler involved.
* Google's protobuf library does have a dynamic/reflection option that avoids my previous point, but it is ~10x slower than generating C++ code. My library, last time I benchmarked it, was 70-90% of the speed of generated C++.