Elasticsearch node crashes can cause data loss

Re:Elasticsearch node crashes can cause data loss by eclark on Saturday May 02, 2015 @12:16AM (#9475850)

Crashes of a program will not affect the data being written to disk if said data has been written into the FS cache (not using std::ostream::write or other in user space buffering). Dirty pages will eventually be written to disk even if the process dies un-cleanly. Only something that keeps the kernel from flushing to disk can keep the page from being eventually written out. ( driver bug, kernel bug, hardware failure, power failure ).

From reading the code in Jepsen it looks like kill -9 is all that's being used to start failures. So there's a real bug here: https://github.com/aphyr/jepsen/blob/master/elasticsearch/sr...

[ Parent | on HN ]

Re:Elasticsearch node crashes can cause data loss by rdtsc on Saturday May 02, 2015 @12:32AM (#9475879)

I think Kyle was just going by the documentation. And that is often what he tests -- how does the reality compare to the claims in the documentation and marketing.

So given these claims:

> Per-Operation Persistence. Elasticsearch puts your data safety first. Document changes are recorded in transaction logs on multiple nodes in the cluster to minimize the chance of any data loss.

One would hope they at least flushed the user space buffers.

[ Parent | on HN ]