kimchy

Qa

actually, it doesn't fixes the mentioned bug. I am only posting this here to make sure people won't go and try and install it thinking it does...

There is a difference between cluster level master election and replication semantics in Elasticsearch (and other similar systems). Even if you use Zookeeper for cluster level master election, one still needs to handle point to point replication (which doesn't go through zk).

grafana is a fork of Kibana. Kibana is a visualization tool of data in Elasticsearch, and Marvel is based on Kibana.

So no, its the other way around.

Marvel is not a "big enterprise" play, its an effort to build a product that applies to a broad user base of Elasticsearch, a product that stands on merit and gives concrete value back to any user in helping them manage and monitor Elasticsearch in production. Because it applies to such a large user base, we decided to make it super affordable so anybody should be able to allow it.

Obviously, we are still providing production support subscriptions for Elasticsearch, Logstash and Kibana, and have been providing it for a long some time.

Marvel works with 0.90.9 and above, which I highly suggest you upgrade from 0.90.5 if you are on it in any case.

Actually, you can use Elasticsearch, Logstash and Kibana today for monitoring. There are many users that happily do that, for example, use Logstash with collectd input, store it in Elasticsearch, and build custom dashboards in Kibana. We built Marvel on top of this open source foundation.

NRT is just part of the story, since in order to provide a proper (near) real time features in a distributed architecture you need to build it around it. For example, not doing pull replication, but push replication. The whole distributed nature of elasticsearch compared to Solr was not discussed in this blog post, but its a big differentiator.

If you are looking to build something similar, check out the open source elasticsearch (http://www.elasticsearch.com). It was built with the mentioned architecture in mind (sorry for the "plug", I am the creator of elasticsearch).

ElasticSearch sharding is implemented in a manner that is on par with current, state of the art, data grids, and probably much better than most nosql solutions out there, but I am biased, since I wrote it... .

For example, elasticsearch is fully API driven, and you can dynamically created indices on it using an API. When you create an index, you can control its sharding and number of replicas, and elasticsearch will take care of the rest. Just start more nodes, and shards will be reallocated. Execute a request to a node, and it will automatically either route the request to the correct node, or do a map / reduce on the relevant shards to perform a search.

With its auto discovery, its a snap to start it. No need to preconfigure anything, especially with multicast hostile cloud environments, since its smart enough to use the cloud API to discover other nodes (see here: http://www.elasticsearch.com/blog/2010/05/11/here-comes-the-...).

Of course, there are much more features to elasticsearch then just being distributed. For example, it can "speak" the application domain model: http://www.elasticsearch.com/blog/2010/02/12/yourdatayoursea..., and has smart long term persistance support (ala data grid write behind support) which includes cloud blob stores (S3, CloudFiles) and hadoop (http://www.elasticsearch.com/blog/2010/02/16/searchengine_ti...). There is a nice high level breakdown of the features here: http://www.elasticsearch.com/products/elasticsearch/.