Show HN: Homogenius – Packer/unpacker to reduce the size of JSON

Re:Show HN: Homogenius – Packer/unpacker to reduce the size ... by walrus on Sunday September 14, 2014 @06:00AM (#8314903)

In most real-world cases, it'll look like this:

  gzip+Homogenius < gzip < Homogenius < original JSON

gzip+Homogenius will typically be better than just gzip because Homogenius 'knows' about the structure of JSON. gzip will typically be better than Homogenius (without gzip) because it can compress the payload.

Here's the size (in bytes) of the sample files given with the project:

   69934 packed-homogenius.json.gz
   79550 verbose.json.gz
  213742 packed-homogenius.json
  265177 verbose.json

[ Parent | on HN ]

Re:Show HN: Homogenius – Packer/unpacker to reduce the size ... by _delirium on Sunday September 14, 2014 @11:05AM (#8315323)

I get the same ordering with gzip, but it seems sensitive to compression algorithm. With bzip2 (default settings, v1.0.6), homogenius seems to actually worsen the compression ratio, at least on this one example:

     52093 verbose.json.bz2
     56298 packed-homogenius.json.bz2
     68521 packed-homogenius.json.gz
     79222 verbose.json.gz
    213742 packed-homogenius.json
    265177 verbose.json

[ Parent | on HN ]

Re:Show HN: Homogenius – Packer/unpacker to reduce the size ... by Scaevolus on Sunday September 14, 2014 @02:10PM (#8315815)

BWT-based compressors like bzip2 do best when their inputs have highly repetitive structure. In a JSON file with many repeated keys, the information that `"k` is usually followed by `ey_1": "` is compressed very effectively. For similar reasons, bzip2 tends to outperform gzip on executables-- it can more effectively model the conditional probability of opcode sequences.

See http://mattmahoney.net/dc/dce.html#Section_55 for more discussion

[ Parent | on HN ]

Re:Show HN: Homogenius – Packer/unpacker to reduce the size ... by beefsack on Sunday September 14, 2014 @06:05AM (#8314907)

You had the same idea as me, I think I posted my edit almost exactly when you posted this!

Interesting to see a similar correlation in your results in the ratio between the two non-gzipped and gzipped files.

[ Parent | on HN ]