A hot/cold tier architecture might work for setups that have a predictable and sharp cutoff from hot to cold. E.g. a logging setup, where the current days index gets all the indexing and queries. And there's a low concurrency occasional query for cold data

Our workload is rather different from this. Indexing and document updates occur in an somewhat exponential decay pattern back in time, same with queries. So there's less sharp cut offs

If we ran a hot-cold architecture we'd get a few issues

Within each tier we'd get imbalanced workload over the nodes. Since within the tier the workload varies greatly with age of the indexes

We use AWS i3 NVMe SSD instances. d2's with HDDs or using EBS have too long IO latency/throughput/iops even for our "cold" data workload. So a cold tier would scale based on storage needs, but in this tier we'd be wasting lots of compute capacity. And a hot tier would scale based on compute needs, and waste tons of storage capacity.

By running both hot and cold workloads on the same set of nodes we get much more cost effective utilization. Since the hot workload uses most of the aggregate compute capacity, and the cold data uses most of the storage.

But this then necessitates using Shardonnay to ensure we spread workload optimally across the clusters. And the more evenly we can spread it, the higher total utilization we can put on the clusters without having single nodes overload.

A hot/cold architecture would much more costly for our workload. Since we'd have to unused storage on the hot tier, and unused compute capacity on the cold tier. A single tier just makes much more sense for our particular use case

We tried the hot/cold architecture as well, and used it before in our data centre architecture.

But currently we have concluded that it is not worth the added complexity. But that might change again if/when we learn more or get new requirements.

We do change the number of replicas for recent data vs old data. We actually have a 4 tiered approach to how many replicas we use.

Thanks for the article! We have a tiny hosted ELK cluster for our app logging, but we don't have much expertise in good cluster design. We recently implemented monthly rolling indices, and currently have a single 'archive' index for the old data (from the past two years) sharded into 25GB chunks. Would it make sense from a query performance perspective to add extra replicas to the 'archive' shards, to help spread the load between the nodes? Thanks for any thoughts!

Hot/Warm architecture is nice, specially for logging setups. But it requires quite a bit of supporting logic (usually scripts) which is not available on ES itself.