News for hackers. Stuff that matters.
Re:FastDoubleParser: Java port of Daniel Lemires fast_double... by jeffbee on Monday March 22, 2021 @01:27PM (#26543807)
It's been fascinating to watch these developments because in my career I've never encountered a performance-sensitive application that wasn't fixed by just eliminating the parsing of doubles from text format. Parsing them faster never seemed to be a good choice. On the other hand producing human-readable doubles (for debug logs or whatever) is a pretty frequent bottleneck that has also seen great improvements recently (Grisu, then Ryu).

Anyway, it's all great programming but it seems to me a symptom or a larger social failure, where we are making bad protocols faster instead of making the protocols better.

Re:FastDoubleParser: Java port of Daniel Lemires fast_double... by jcelerier on Monday March 22, 2021 @02:14PM (#26544530)
Have you never been in projects where a hard requirement was "save files must be editable with notepad.exe" ? If so, lucky you
Re:FastDoubleParser: Java port of Daniel Lemires fast_double... by kstenerud on Monday March 22, 2021 @11:58PM (#26550933)
It is putting lipstick on a pig, but the problem is that we've gotten used to the human convenience that text-based formats offer (HTTP, HTML, XML, JSON, etc), and we don't want to give them up despite the energy wastefulness of parsing them.

That's why I'm building Concise Encoding [1], which is a twin text/binary format that can be 1:1 converted back and forth. You do everything in binary, and only convert to/from text in the few places where a human needs to be involved (like debugging or configuration).

[1] https://concise-encoding.org

Re:FastDoubleParser: Java port of Daniel Lemires fast_double... by dodobirdlord on Tuesday March 23, 2021 @02:24AM (#26551702)
This format seems to have a lot of parallels to Amazon’s Ion format, though I’m surprised to see a strongly typed binary encoding with 19 types! Definitely not the 16 I’d expect. Is a whole byte used to store the type information, or is something more complex going on?
Re:FastDoubleParser: Java port of Daniel Lemires fast_double... by kstenerud on Tuesday March 23, 2021 @03:34AM (#26552101)
Slightly more complex. The type code distribution is based on how often the type is likely to occur in your average document.

The type field is almost always 1 byte long, except for typed arrays (which would generally in aggregate be long enough to offset the cost) and two uncommon types/variations. These have an effective 2-byte type field (0x94 selects a secondary type plane, and the next byte selects the specific type from there).

The most common integer values (-100 to 100) are encoded directly into the types 0x9c to 0x64 (wraparound), such that interpreting them directly as signed 8-bit signed integers yields their actual value (type 0x00 = integer 0, type 0x64 = integer 100, type 0x9c = -100, etc).

Strings are also optimized such that types 0x80 - 0x8f are used for the most common string lengths (0 to 15) so as not to require a separate length field.

The rest have a 1-byte type field. There are also 3 reserved type codes left in the first plane in case something big comes up in the future. You can see the chart here: https://github.com/kstenerud/concise-encoding/blob/master/cb...

Re:FastDoubleParser: Java port of Daniel Lemires fast_double... by anarazel on Monday March 22, 2021 @08:46PM (#26549528)
Database bulk load are where I have hit this many times. Often it's not viable to change the import to be a binary format (e.g cross databases, migrating from old proprietary software, etc).