Re:How Discord Indexes Billions of Messages Using Elasticsearch
by jhgg on Thursday March 16, 2017 @04:15PM
(#13888760)
ES can indeed rebalance shards to new nodes as they join the cluster. More here: https://www.elastic.co/guide/en/elasticsearch/reference/curr...
Re:How Discord Indexes Billions of Messages Using Elasticsearch
by jdc0589 on Thursday March 16, 2017 @04:16PM
(#13888773)
huh, the more you know. not sure how I've missed this. I'll definitely look for some discussions/use cases of how this actually plays out.
Re:How Discord Indexes Billions of Messages Using Elasticsearch
by jazoom on Thursday March 16, 2017 @04:53PM
(#13889066)
I suspect you were thinking of changing the number of shards, which can be an important part of rebalancing an index. Unfortunately, Elasticsearch can't do that.
Re:How Discord Indexes Billions of Messages Using Elasticsearch
by jonaf on Thursday March 16, 2017 @06:19PM
(#13889708)
There is a convenient reindex API available in more recent versions, although in most cases where you can rebuild your index anyway, it's faster to do that.
Re:How Discord Indexes Billions of Messages Using Elasticsearch
by jazoom on Thursday March 16, 2017 @06:27PM
(#13889753)
Ahh. I was wondering when they were going to get around to implementing that. I might have to upgrade to 5.x soon.
Re:How Discord Indexes Billions of Messages Using Elasticsearch
by wsgolfer on Thursday March 16, 2017 @07:34PM
(#13890121)
FYI, the reindex API was added as experimental in 2.3, so it's been there for about a year [1]. Even as experimental it worked great.
[1] https://www.elastic.co/guide/en/elasticsearch/reference/2.3/...
Re:How Discord Indexes Billions of Messages Using Elasticsearch
by jazoom on Thursday March 16, 2017 @07:41PM
(#13890152)
Ha. I just came here to edit my comment. I remember seeing it before and just checked the docs again. In 5.2 it's STILL considered experiential. Craziness. It took me about an hour to build The basic functionality myself.
Re:How Discord Indexes Billions of Messages Using Elasticsearch
by jasontedor on Thursday March 16, 2017 @10:10PM
(#13890810)
Thank you for noticing this, it is not considered experimental anymore. I integrated a change[1] to update the docs; it should percolate through to the site within thirty minutes.
Re:How Discord Indexes Billions of Messages Using Elasticsearch
by jdc0589 on Thursday March 23, 2017 @11:07AM
(#13940362)
FYI, the re-index API is awesome.
Re:How Discord Indexes Billions of Messages Using Elasticsearch
by jazoom on Thursday March 16, 2017 @11:25PM
(#13891174)
Great work. It appears to have already percolated.
Re:How Discord Indexes Billions of Messages Using Elasticsearch
by packetized on Thursday March 16, 2017 @06:01PM
(#13889610)
Tuning cluster rebalance settings can be... challenging. I've done some bad, bad things with a cluster where it would effectively balance things back and forth - forever.
Fiddling with cluster.routing.allocation.balance.* can really paint you into a corner quickly.
Re:How Discord Indexes Billions of Messages Using Elasticsearch
by pram on Thursday March 16, 2017 @05:22PM
(#13889326)
Yes that's actually one of my favorite ES features. Kafka doesn't do any type of automatic topic partition rebalancing which was quite shocking to me when I first started using it. Seems kind of primitive in comparison.
I'm assuming they mean "I can add a new node to the cluster, and NEW SHARDS FROM NEW INDEXES can be distributed amongst the new nodes".... So far as I know elasticsearch can't rebalance shard location or composition automatically based on cluster membership change events...right? I mean, the cluster reroute api will let you manually move a shard, but thats all I know of.