News for hackers. Stuff that matters.
Re:JCOF: JSON-like Compact Object Format by bearjaws on Friday July 15, 2022 @01:20PM (#32110545)
People keep reinventing JSON trying to beat gzip, and it never beats the simplicity of gzipping JSON...
Re:JCOF: JSON-like Compact Object Format by xsmasher on Friday July 15, 2022 @02:01PM (#32111037)
comets.json:

        JSON: 51949 bytes
        JCOF: 37480 bytes (0.721x)
  JSON ZIPED: 15178 bytes
Zipped json wins again.
Re:JCOF: JSON-like Compact Object Format by conradev on Friday July 15, 2022 @02:29PM (#32111316)
Now I am curious – how does zipped JCOF turn out? I know that’s not the author’s intention, but I’m curious how it compresses
Re:JCOF: JSON-like Compact Object Format by bearjaws on Friday July 15, 2022 @03:18PM (#32111785)
It's nearly irrelevant, given that I can gzip JSON and send it to literally anything built in the last 10+ years. It's so ubiquitous now that many web servers will just accept gzipped content without even exposing that fact to the back end servers. You can get all the benefits of highly compressed JSON and never have interacted with gzip.

If it could somehow produce a significant reduction with fewer CPU cycles, there may be a really niche use case, but I don't see those often.

Re:JCOF: JSON-like Compact Object Format by conradev on Friday July 15, 2022 @05:24PM (#32112971)
Oh, it is completely irrelevant! I just want a data point for my mental model of gzip performance. Specifically, I am curious if the optimizations (object tables and separator elimination) are made irrelevant by gzip's dictionary, or if they result in better compression
Re:JCOF: JSON-like Compact Object Format by twawaaay on Friday July 15, 2022 @05:43PM (#32113129)
In my experience it should not make much or any difference.

I had devs try to "optimise" the JSON our services were returning by doing things like abbreviating field names that were repeated very often or replacing long string constants with short integers or removing formatting, and so on. When I asked them to do actual test when the service returns gzipped output and they found out that even reducing the JSON size this way by half does not make any measurable difference on the compressed stream in most cases.

Re:JCOF: JSON-like Compact Object Format by dreamdeath on Saturday July 16, 2022 @09:56AM (#32118121)
I found the same re: compressed stream, however I also found that hand optimizing the decompressed JSON made for significant performance improvements on underpowered devices (e.g. some old Android phone).
Re:JCOF: JSON-like Compact Object Format by twawaaay on Sunday July 17, 2022 @05:27PM (#32131503)
Well, that's true. But if performance is more important to you, just skip it and go for binary formats that can be orders of magnitude more efficient.

In general text formats are a compromise. They are bulky and and inefficient to parse. What you get in exchange is ease of development.

If you are willing to complicate the format just to improve performance, at some point you no longer get ease of development and you should just switch to binary altogether.

Re:JCOF: JSON-like Compact Object Format by mike_hock on Friday July 15, 2022 @05:40PM (#32113109)
And zipped/unzipped msgpack for that matter.
Re:JCOF: JSON-like Compact Object Format by petre on Sunday July 17, 2022 @10:07AM (#32126882)
Precisely, I'd rather use msgpack or CBOR if it requires additional libs. I use JSON because of browser support.
Re:JCOF: JSON-like Compact Object Format by Beltiras on Friday July 15, 2022 @01:39PM (#32110784)
This actually does. I used Python gzip to see how the minified would compress and came up with 157 bytes to JCOF's 134. Using the same mechanism to gzip the JCOF saves only 4 bytes. I'm actually rather impressed.
Re:JCOF: JSON-like Compact Object Format by zimpenfish on Friday July 15, 2022 @02:17PM (#32111198)
It might on small files but it doesn't on (some) bigger ones. e.g. I dumped a Minecraft region file into JSON giving 85M. `jcof` turns this into 66M. `gzip` on the original json gives 14M for `-1` and 11M for `-9`. `zstd` also gives 11M.

But because the Minecraft JSON is highly redundant, I also tried it on 84M of fake social graph data. `jcof` gave 44M, `gzip -9` gave 19M and `zstd` gave 18M.

In summary, neat idea, but you're maybe better off with `gzip` or `zstd` in the real world.

Re:JCOF: JSON-like Compact Object Format by Beltiras on Friday July 15, 2022 @02:22PM (#32111248)
JCOF and gzip are not mutex.
Re:JCOF: JSON-like Compact Object Format by masklinn on Friday July 15, 2022 @02:55PM (#32111581)
No, but when you get ~95% of the benefit (per other people's data) while keeping all the flexibility, compatibility with everything under the sun, and native-performance parsing everywhere, the supposed improvement looks a lot less like one.
Re:JCOF: JSON-like Compact Object Format by Beltiras on Friday July 15, 2022 @04:51PM (#32112676)
A sibling has the data. For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits. I think this tool is a nice one to have in the toolbox. The low-rate files seem to be close to optimal.
Re:JCOF: JSON-like Compact Object Format by KronisLV on Saturday July 16, 2022 @05:56PM (#32122334)
> For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits.

Provided that you don't run into the issue of one end only talking in language X, which does not have JCOF support due to there not being any libraries for it yet. In the GitHub repo, it seems like there is only the JavaScript reference implementation for now.

Re:JCOF: JSON-like Compact Object Format by Beltiras on Saturday July 16, 2022 @06:23PM (#32122481)
The design documents are really thorough. I don't think I would be more than a day making a serializer in Python for example. Lack of libraries just means an interested developer needs to tend to the matter and solve the issue.
Re:JCOF: JSON-like Compact Object Format by zimpenfish on Friday July 15, 2022 @05:26PM (#32112992)
> A sibling has the data.

And now has updated data showing that sorting your keys lets `lzma` beat `jcof|lzma` at every level. The egress bill has been reduced! All rejoice!

Re:JCOF: JSON-like Compact Object Format by zimpenfish on Friday July 15, 2022 @07:53PM (#32113981)
EDIT: Nope, I was wrong, I was doing `lzma` against `jcof|zstd`. `jcof|lzma` is still sneaking ~1M below `lzma` at all levels.
Re:JCOF: JSON-like Compact Object Format by luhn on Friday July 15, 2022 @03:47PM (#32112032)
You just blew my mind that “mutex” is a portmanteau of “mutually exclusive.” How have I been programming for 20 years and never realized this?
Re:JCOF: JSON-like Compact Object Format by Beltiras on Friday July 15, 2022 @04:47PM (#32112639)
Perhaps you haven't had to program concurrencies or thought much about bitwise operators. It's not very frequent anymore. Used to be.
Re:JCOF: JSON-like Compact Object Format by zimpenfish on Friday July 15, 2022 @03:13PM (#32111748)
See https://news.ycombinator.com/item?id=32111239 where I tested gzip vs jcof vs jcof+gzip (tl;dr: you save 2M at the cost of an extra CPU intensive process)
Re:JCOF: JSON-like Compact Object Format by tmp_anon_22 on Friday July 15, 2022 @02:05PM (#32111078)
I think people would reject your samples because the data size is too small. Production JSON comes in all shapes and sizes and situations where you care about performance usually indicates much larger payloads.
Re:JCOF: JSON-like Compact Object Format by Beltiras on Friday July 15, 2022 @04:49PM (#32112656)
If your egress is paramount, you should of course benchmark everything to shreds. Then pick what fits.
Re:JCOF: JSON-like Compact Object Format by heresie-dabord on Friday July 15, 2022 @03:15PM (#32111766)
People keep trying to re-invent a human-readable, parsable, efficient, robust format that can be validated.

But "easy" wins so we keep returning to CSV and JSON. ^_^

Re:JCOF: JSON-like Compact Object Format by nine_k on Friday July 15, 2022 @04:47PM (#32112642)
Gzip may be standard and supported transparently by your http library.

OTOH zstd [1] should be significantly more efficient, while being as fast or faster.

[1]: https://en.wikipedia.org/wiki/Zstd