News for hackers. Stuff that matters.
Re:Fast function to parse strings into double (binary64) flo... by userbinator on Tuesday March 10, 2020 @10:47PM (#22542479)
HN (and a lot of other places, really) has a large population of web or otherwise "web-heritage" developers, as well as those whose ventures into computing started beside/before the web. I suspect the JSON promotion comes mostly from the former group, while the latter, of which I am a member, laments it roughly as much as we did the "XML craze" of the late 90s and early 2000s.
Re:Fast function to parse strings into double (binary64) flo... by xxs on Wednesday March 11, 2020 @08:59AM (#22545409)
Indeed, hand rolled fixed point network marshaling, preferably with delta encoding for time series alike data (or data that oscillates) is couple of orders of magnitude faster and but also significantly "smaller" than a double-string-double, counterpart. Getting ~1 byte per data point of time series is quite achievable.
Re:Fast function to parse strings into double (binary64) flo... by vbezhenar on Wednesday March 11, 2020 @09:17AM (#22545545)
How do you encode floating point delta into one byte?
Re:Fast function to parse strings into double (binary64) flo... by thedance on Wednesday March 11, 2020 @12:56PM (#22547375)
You can't, that's why they said fixed point. Time series data should probably never use floats, for the exact reason that delta-encoding is not effective.
Re:Fast function to parse strings into double (binary64) flo... by scottlamb on Wednesday March 11, 2020 @05:50PM (#22551173)
"Can't" is too strong a word. Depending on what your numbers represent, I'd think you could do one or more of:

* have a shorter exponent and/or mantissa to make each number shorter and the overall series more repetitive. (maybe even skip the mantissa if you only care about order of magnitude! call it the opposite of fixed-point, or just storing the logarithm of the value, whatever terminology works for you...)

* skip the sign bit for a non-negative series (e.g., deltas of a non-decreasing series),

* use a variable-width bitstream encoding like exp-Golomb to represent the exponent and/or mantissa,

* use run-length encoding if it's repetitive (after delta-encoding),

* etc.

One byte per value (or less!) is surely possible in some cases. But I've never done it myself and agree fixed-point sounds much more pleasant. One of my projects represents deltas of durations in units of 90,000ths of a second, varint-encoded. I much prefer that to dealing with float seconds.

Re:Fast function to parse strings into double (binary64) flo... by thedance on Wednesday March 11, 2020 @07:40PM (#22551988)
There's also the XOR compression scheme described by Facebook:

https://www.vldb.org/pvldb/vol8/p1816-teller.pdf

Re:Fast function to parse strings into double (binary64) flo... by cm2187 on Wednesday March 11, 2020 @05:05AM (#22544250)
And a tiny minority I am proudly part of, who watches these debates about xml vs json or tabs vs whitespace with the same amusement than Mike Judge!
Re:Fast function to parse strings into double (binary64) flo... by xxs on Wednesday March 11, 2020 @09:00AM (#22545417)
C'mon spaces vs tab is a serious one and it makes sense... unlike xml, json shizz.
Re:Fast function to parse strings into double (binary64) flo... by saagarjha on Wednesday March 11, 2020 @09:02AM (#22545434)
Spaces versus tabs is a stupid debate because "tabs for indentation, spaces for alignment" is clearly the correct answer.