News for hackers. Stuff that matters.

A comment on Fast key-value stores: An idea whose time has come and gone. Parent comment

Re:Fast key-value stores: An idea whose time has come and gone by shereadsthenews on Monday June 24, 2019 @02:00PM (#20266434)
For simple protocols protobuf decoding has no taken branches. I.e. if you only use the first 15 field numbers (all your tags are 1 byte) and if all the types are the expected types, and if all the variable-length items are < 128 bytes long then you can decode the message without taking any branches. In C++. Most of the other languages have simpler and slower codecs.

This is the hot path in C++[1]. A really large amount of work has gone into protobuf C++ performance in the last 3 years or so.

1: https://github.com/protocolbuffers/protobuf/blob/master/src/...

[ Parent | on HN ]
Re:Fast key-value stores: An idea whose time has come and gone by kentonv on Monday June 24, 2019 @02:12PM (#20266565)
And all your integer fields must be < 128, right?

Yes, I suppose the branches in Protobuf can be pretty predictable. Still, you do generally have to examine each byte individually.

[ Parent | on HN ]
Re:Fast key-value stores: An idea whose time has come and gone by shereadsthenews on Monday June 24, 2019 @02:30PM (#20266778)
Sure. In this specific case of a kv store it's hard to imagine how to simplify it dramatically from protobuf. As a proto you might have: tag-length-key-tag-length-value. Instead you could store the key and value lengths in host format using 8-16 bytes: length-length-key-value. It's not _dramatically_ faster to decode this, and you traded away extensibility to get a marginal speedup.
[ Parent | on HN ]
Re:Fast key-value stores: An idea whose time has come and gone by kentonv on Monday June 24, 2019 @02:54PM (#20267069)
Sure, I was speaking in general, not specifically about the key-value case.

I think most serialization frameworks are likely to be overkill for such a use case, spending more time on setup than actual parsing.

Also note that storing the value (and maybe the key) with proper alignment might make it easier to use the data in-place, saving a copy.

[ Parent | on HN ]
Re:Fast key-value stores: An idea whose time has come and gone by yencabulator on Saturday June 29, 2019 @08:05PM (#20315612)
Hit 'y' before copying the link; the line numbers have already shifted.
[ Parent | on HN ]