News for hackers. Stuff that matters.
Re:MessagePack: Fast and Compact Serialization by jstephens on Tuesday August 16, 2011 @05:16PM (#2892920)
For perl it's fast as long as you have a small payload. Here's my benchmarks using progressively bigger data structures.

    perl: 5.008008
    Storable: 2.21
    JSON::XS: 2.25
    Data::MessagePack: 0.34

    ==== Size ====

    .-----------------+-----------+---------+---------.
    | src             | storable  | json    | msgpack |
    +-----------------+-----------+---------+---------+
    | 1               | 4         | fail    | 1       |
    | 3.14            | 8         | fail    | 5       |
    | {}              | 7         | 2       | 1       |
    | []              | 7         | 2       | 1       |
    | [('a')x10]      | 37        | 41      | 21      |
    | {('a')x10}      | 14        | fail    | 11      |
    | +{1,+{1,+{}}}   | 29        | 14      | 7       |
    | +[+[+[]]]       | 19        | 6       | 3       |
    '-----------------+-----------+---------+---------'
    ==========================

    Testing: theirs:HASH (4995 bytes)

    ==== Serialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.14 usr +  0.00 sys =  1.14 CPU) @ 26947.37/s (n=30720)
            mp:  1 wallclock secs ( 1.11 usr +  0.00 sys =  1.11 CPU) @ 70446.85/s (n=78196)
      storable:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 18442.45/s (n=19549)
                Rate storable     json       mp
    storable 18442/s       --     -32%     -74%
    json     26947/s      46%       --     -62%
    mp       70447/s     282%     161%       --

    ==== Deserialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.02 usr +  0.00 sys =  1.02 CPU) @ 23424.51/s (n=23893)
            mp:  1 wallclock secs ( 1.02 usr +  0.00 sys =  1.02 CPU) @ 35136.27/s (n=35839)
      storable:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 25357.55/s (n=26879)
                Rate     json storable       mp
    json     23425/s       --      -8%     -33%
    storable 25358/s       8%       --     -28%
    mp       35136/s      50%      39%       --


    ==========================

    Testing: xdbi:HASH (24138 bytes)

    ==== Serialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 2414.15/s (n=2559)
            mp:  1 wallclock secs ( 1.07 usr +  0.00 sys =  1.07 CPU) @ 8373.83/s (n=8960)
      storable:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 7244.34/s (n=7679)
               Rate     json storable       mp
    json     2414/s       --     -67%     -71%
    storable 7244/s     200%       --     -13%
    mp       8374/s     247%      16%       --

    ==== Deserialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.11 usr +  0.00 sys =  1.11 CPU) @ 4842.34/s (n=5375)
            mp:  2 wallclock secs ( 1.13 usr +  0.00 sys =  1.13 CPU) @ 4324.78/s (n=4887)
      storable:  1 wallclock secs ( 1.05 usr +  0.00 sys =  1.05 CPU) @ 4654.29/s (n=4887)
               Rate       mp storable     json
    mp       4325/s       --      -7%     -11%
    storable 4654/s       8%       --      -4%
    json     4842/s      12%       4%       --


    ==========================

    Testing: pdf:HASH (35239 bytes)

    ==== Serialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.04 usr +  0.00 sys =  1.04 CPU) @ 9845.19/s (n=10239)
            mp:  1 wallclock secs ( 1.05 usr +  0.00 sys =  1.05 CPU) @ 5119.05/s (n=5375)
      storable:  1 wallclock secs ( 1.10 usr +  0.00 sys =  1.10 CPU) @ 7518.18/s (n=8270)
               Rate       mp storable     json
    mp       5119/s       --     -32%     -48%
    storable 7518/s      47%       --     -24%
    json     9845/s      92%      31%       --

    ==== Deserialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.08 usr +  0.00 sys =  1.08 CPU) @ 3110.19/s (n=3359)
            mp:  1 wallclock secs ( 1.07 usr +  0.00 sys =  1.07 CPU) @ 2790.65/s (n=2986)
      storable:  1 wallclock secs ( 1.08 usr +  0.00 sys =  1.08 CPU) @ 3554.63/s (n=3839)
               Rate       mp     json storable
    mp       2791/s       --     -10%     -21%
    json     3110/s      11%       --     -13%
    storable 3555/s      27%      14%       --


    ==========================

    Testing: globallib:HASH (197319 bytes)

    ==== Serialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.09 usr +  0.00 sys =  1.09 CPU) @ 1760.55/s (n=1919)
            mp:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 791.51/s (n=839)
      storable:  1 wallclock secs ( 1.05 usr +  0.00 sys =  1.05 CPU) @ 1279.05/s (n=1343)
               Rate       mp storable     json
    mp        792/s       --     -38%     -55%
    storable 1279/s      62%       --     -27%
    json     1761/s     122%      38%       --

    ==== Deserialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.71 usr +  0.00 sys =  1.71 CPU) @ 301.75/s (n=516)
            mp:  1 wallclock secs ( 1.03 usr +  0.00 sys =  1.03 CPU) @ 404.85/s (n=417)
      storable:  1 wallclock secs ( 1.12 usr +  0.00 sys =  1.12 CPU) @ 544.64/s (n=610)
              Rate     json       mp storable
    json     302/s       --     -25%     -45%
    mp       405/s      34%       --     -26%
    storable 545/s      80%      35%       --
Re:MessagePack: Fast and Compact Serialization by dimva on Monday August 15, 2011 @09:57AM (#2886902)
In node.js, JSON is actually faster than msgpack: https://github.com/aikar/wormhole/issues/3

Also, due to V8's poor C++ API, msgpack doesn't support deserialization of 64 bit integers.

Re:MessagePack: Fast and Compact Serialization by mraleph on Monday August 15, 2011 @04:59PM (#2888680)
> Also, due to V8's poor C++ API, msgpack doesn't support deserialization of 64 bit integers.

JavaScript _does_not_ have a number type wide enough to hold 64-bit integer. V8 API has nothing to do with it.

Re:MessagePack: Fast and Compact Serialization by Ralith on Monday August 15, 2011 @09:01AM (#2886648)
This sounds an awful lot like yet another ProtocolBuffers. What, if anything, new does it bring to the table?
Re:MessagePack: Fast and Compact Serialization by ericmoritz on Monday August 15, 2011 @10:15AM (#2886965)
This is schemaless. Protobufs require a .proto file that describes the data being serialized. A .proto file serves a similar purpose as the `struct` keyword in C.

MessagePack uses type prefixes to describe the data. For instance an unsigned 8-bit int starts with a `0xcc` followed by a byte.

Re:MessagePack: Fast and Compact Serialization by frsyuki on Monday August 15, 2011 @11:47AM (#2887353)
Compatibility with JSON is a characteristic of MessagePack. If you're already using JSON, you can use MessagePack as a faster/smaller substitution of JSON.

Since Protocol Buffers brings their original semantics, you must change lots of codes.

Re:MessagePack: Fast and Compact Serialization by frsyuki on Monday August 15, 2011 @11:32AM (#2887284)
Schemaless (dynamic typing) is the most obvious difference.

I wrote about the pros/cons of schema here: http://news.ycombinator.com/item?id=2836242

Re:MessagePack: Fast and Compact Serialization by xal on Monday August 15, 2011 @08:45AM (#2886587)
We use this as data exchange format for our RPC services at Shopify. Incredibly fast, stable, easy to use.
Re:MessagePack: Fast and Compact Serialization by mdonahoe on Monday August 15, 2011 @08:54AM (#2886619)
Is this actually better than gzipped JSON?
Re:MessagePack: Fast and Compact Serialization by rabidsnail on Monday August 15, 2011 @01:53PM (#2887924)
This appears to be mmap-friendly. Since arrays have length values it's possible to jump around in a large messagepack-serialized structure without having to scan it. In JSON if you want to get to element 2 of an array you have to scan to the end of element 1. If there's a length value at the beginning of element 1 (as it looks like there is with msgpack) then you can seek straight past it.
Re:MessagePack: Fast and Compact Serialization by roel_v on Monday August 15, 2011 @09:24AM (#2886744)
Depends on how you define 'better', I guess - at the very least it has to be several magnitudes faster, with no overhead for compression, parsing, etc.
Re:MessagePack: Fast and Compact Serialization by frsyuki on Monday August 15, 2011 @12:00PM (#2887414)
You can use type-conversion APIs with MessagePack if you use C++, Java or C#. It converts deserialized objects (=dynamically-typed) into statically-typed objects using templates (C++) or reflection + dynamic code generation (Java and C#). You don't have to write boring type-checking codes.

Note that this is a characteristic of the implementation, not of the data format. Some libraries for JSON like JSONIC (Java) provides these APIs.

Re:MessagePack: Fast and Compact Serialization by zdw on Monday August 15, 2011 @09:38AM (#2886816)
Does it have it suitably low memory/CPU requirements to run on embedded devices?

How well does it work on ARM/MIPS?

Re:MessagePack: Fast and Compact Serialization by frsyuki on Monday August 15, 2011 @11:19AM (#2887235)
It works very well at least on ARM.

I've used MessagePack on iPhone 3GS for the format of a dictionary file (> 100MB). I utilized mmap and zero-copy deserialization feature of MessagePack. And I received (and merged) some patches sent by those who use it on iPhone4/iPad2.

I've never tried to run it on MIPS. But it probably works if you use gcc.

(I'm author of MessagePack)

Re:MessagePack: Fast and Compact Serialization by erluser on Monday August 15, 2011 @12:17PM (#2887493)
Would be nice if suuport for RPC (and generally) for Erlang was better. It's "currently" in alpha.