frsyuki

ACK of network transfer is available ("require_ack_response" option). This option ends up choice of at-most-once semantics vs. at-least-once semantics. You need to choose and you can choose.

Fuentd provides "buffer_type file" to buffer records on disk. Shutting down won't loose data. If you need to choose memory buffer for performance reasons, fluentd enables "flush_at_shutdown" option by default.

You would also want to use <secondary> feature. This lets you to write a buffer chunk to another storage if the primary destination is not available "retry_limit" times.

Those concerns would be solved by the document: http://docs.fluentd.org/articles/out_forward#buffered-output...

Did you guys see this video?: http://vimeo.com/53261709 Github uses MessagePack for their internal RPC protocol.

Here is a benchmark result of msgpack, json, bson and protobuf: https://gist.github.com/4348013

An important point we should consider here is that actual performance depends on implementation.

Disclaimer: I'm authoer of MessagePack for C++/Ruby and committer of one for Java.

As for strings, JSON has to allocate memory and copy to deserialize strings because strings are escaped.

MessagePack does't have to allocate/copy because the serialized format of strings is same as the format in memory. But it depends on the implementation whether actually it doesn't allocate/copy.

C++ and Ruby implementations try to suppress allocation and copying (zero-copy). But Java implementation doesn't support zero-copy feature so far (we have plan to do so. Here is "TODO" comment: https://github.com/msgpack/msgpack-java/blob/master/src/main...).

As for the other types, C++ implementation (and new Ruby implementation which is under development) has memory pool which optimizes those memory allocations petterns. But it's hard to implement such optimizations for Java because JVM (and Dalvik VM) doesn't allow to hook object allocation.

We're using MessagePack in a Rails application to log user behaviors and analyze them.

Compared with other serialization libraries such as Protocol Buffers, Avro or BSON, one of the advantages of MessagePack is compatibility with JSON. (In spite of its name, BSON has special types which cause incompatibility with JSON)

It means we can exchange objects sent from browsers (in JSON format) between servers written in different languages without losing information.

I will not use MessagePack with browsers but it's still useful to use it with web applications.

MessagePack includes a concept named "type conversion" to support types which are not supported by its wire format. With the concept, we can serialize/deserialize user-defined classes as well as strings with encodings.

So far, MessagePack for Java, C++ and D implement the concept.

Although the original blog post focuses on JavaScript and browsers, MessagePack itself doesn't mainly focus on them.

A major use case of MessagePack is to store serialized objects in memcached. A blog post written by Pinterest describes this use case (http://engineering.pinterest.com/posts/2012/memcache-games/). They use MessagePack with Python which is faster than one with JavaScript. They could store more objects in a server without performance declination (e.g. gzip).

It's true that MessagePack is not always faster than JSON (e.g. within browsers), and it's not always smaller than other serialization methods (e.g. with gzip compression). So we should consider that which serialization methods should I use for "my" case.

There are also general tendency which is helpful to select MessagePack or JSON:

    MessagePack is faster to serialize binary data such as thumbnail images.
    MessagePack is better to reduce overheads to exchange small objects between servers.
    JSON is better to use it with browsers.

We're also using Fluentd as well as original JSON-based logging libraries.

Fluentd deals with JSON-based logs. JSON is good for human facing interface because it is human readable and GREPable.

On the other side, Fluentd handles logs in MessagePack format internally. Msgpack is a serialization format compatible with JSON and can be an efficient replacement of JSON.

I wrote plugin for Fluentd that send those structured logs to Librato Metrics (https://metrics.librato.com/) which provides charting and dashboard features.

With Fluentd, our logs became program-friendly as well as human-firnedly.

Spitting out sprintfs means parsing texts. JSON might be slow since it's text, but binary based formats should be faster especially the log includes many integers.

There are binary-based serialization formats such as MessagePack [1] or BSON [2] that doesn't need escapeing overhead.

You can simply replace JSON with MessagePack since MessagePack is compatible with JSON.

[1] Why Not MessagePack? http://blog.andrewvc.com/why-arent-you-using-messagepack

[2] Performant Entity Serialization: BSON vs MessagePack (vs JSON) http://stackoverflow.com/questions/6355497/performant-entity...

You can use type-conversion APIs with MessagePack if you use C++, Java or C#. It converts deserialized objects (=dynamically-typed) into statically-typed objects using templates (C++) or reflection + dynamic code generation (Java and C#). You don't have to write boring type-checking codes.

Note that this is a characteristic of the implementation, not of the data format. Some libraries for JSON like JSONIC (Java) provides these APIs.

Compatibility with JSON is a characteristic of MessagePack. If you're already using JSON, you can use MessagePack as a faster/smaller substitution of JSON.

Since Protocol Buffers brings their original semantics, you must change lots of codes.

Schemaless (dynamic typing) is the most obvious difference.

I wrote about the pros/cons of schema here: http://news.ycombinator.com/item?id=2836242

It works very well at least on ARM.

I've used MessagePack on iPhone 3GS for the format of a dictionary file (> 100MB). I utilized mmap and zero-copy deserialization feature of MessagePack. And I received (and merged) some patches sent by those who use it on iPhone4/iPad2.

I've never tried to run it on MIPS. But it probably works if you use gcc.

(I'm author of MessagePack)

(Note: I'm author of MessagePack)

I've described about the difference between MessagePack and BSON:

"Performant Entity Serialization: BSON vs MessagePack (vs JSON)" http://stackoverflow.com/questions/6355497/performant-entity...