I'm a little suprised to learn there's no self-reference support in protobuf, as I wouldn't have assumed parsing that would be an issue (as all it really is is a pointer to an existing object in the message to say, put a ref. to it here), though I guess it might be a problem in supporting certain languages.
That's a tradeoff more designs should have, IMO: reduce the feature set as much as possible, but in return make the implementation vastly simpler. :)
I assume it's not only about support in programming languages, but also exactly to eliminate the entire class of bugs that stems from back/forward-references in serialized data, and to generally keep the wire format as simple (to parse and to implement a parser for) as possible. The few usecases that could make use of references are not worth the pain inflicted on everyone if they were implemented.
What this means is that if your IDL specifies deep nesting (or recursive nesting), then it means your application is expected to handle this “by contract”, and attempts to deserialize will rightfully fail in case of out-of-memory / stack overflow errors. There's no danger of an implementation 'accidentally' deserializing something nested that was passsed from the outside, as anything unknown to the IDL is simply ignored.
Finally, there's no XML-like self-references in Protobuf, so it's not possible to have an infinitely deep structure, or a combinatorial explosion like with billion laughs - just a very deeply nested one, and only if allowed in the IDL, and only up to whatever message size limit you're allowing.