mikljohansson

Wow, looks so nice! I'm going to use the widget for my model cards

Thanks! We have a fair bit of a punny humor in our Gothenburg-Sweden engineering office where the information retrieval and big data magic happens. One of our plugins for Elasticsearch that significantly improves GC pressure and performance is called the "Greased Pig" plugin, because it makes Elasticsearch all slippery so it doesn't get stuck in GC hell ;)

Our query complexity and query response times follow a somewhat exponential pattern, with a lot of quick queries and a long tail of more monstrous queries.

We do have a large number of very complex queries coming in. What we deem to be towards the "simple" end could easily be hundreds of terms, wildcard, near and phrase operators. "Difficult" queries are things with hundreds of thousands of terms or many wildcards within nears/phrases that expand to millions of term combinations.

For Meltwaters customers it's usually important to get both very high recall and precision in the dataset described by a query. Since it's very little about getting a ranked result list and finding a hit (e.g. like what Google does). It's much more about running analytics/dashboards/reports/trends over the dataset delineated by the query, and exactness in the analytics matter a lot to Meltwater customers.

This all makes for complicated queries, to get both high precision and recall for whatever a customer is interested in analyzing. Our sales and support organizations help customers write good queries, and we also use AI systems to generate queries

Our whole dataset exported as compressed JSON is likely not much more than 100TB. The petabytes come from all the index datastructures Elasticsearch/Lucene builds, as well as the high replication factor needed to keep up with the query throughput.

We index a lot of NLP and other enrichments on our documents. This also adds a lot of storage on top of the base text. And like Karl mentioned, we have lots of small social media documents available for analytics (currently 34B, sometime next year upward of 230B). But also 7+ billion documents of news, blogs and other long-text articles indexed, basically 10 years of all news media from the entire world online and available for analytics.

We're building the https://fairhair.ai/ data science platform to allow other companies to access, and run online search and analytics on top of this massive dataset and compute clusters. For example to embed analytics over this dataset into their own SaaS products

A hot/cold tier architecture might work for setups that have a predictable and sharp cutoff from hot to cold. E.g. a logging setup, where the current days index gets all the indexing and queries. And there's a low concurrency occasional query for cold data

Our workload is rather different from this. Indexing and document updates occur in an somewhat exponential decay pattern back in time, same with queries. So there's less sharp cut offs

If we ran a hot-cold architecture we'd get a few issues

Within each tier we'd get imbalanced workload over the nodes. Since within the tier the workload varies greatly with age of the indexes

We use AWS i3 NVMe SSD instances. d2's with HDDs or using EBS have too long IO latency/throughput/iops even for our "cold" data workload. So a cold tier would scale based on storage needs, but in this tier we'd be wasting lots of compute capacity. And a hot tier would scale based on compute needs, and waste tons of storage capacity.

By running both hot and cold workloads on the same set of nodes we get much more cost effective utilization. Since the hot workload uses most of the aggregate compute capacity, and the cold data uses most of the storage.

But this then necessitates using Shardonnay to ensure we spread workload optimally across the clusters. And the more evenly we can spread it, the higher total utilization we can put on the clusters without having single nodes overload.

A hot/cold architecture would much more costly for our workload. Since we'd have to unused storage on the hot tier, and unused compute capacity on the cold tier. A single tier just makes much more sense for our particular use case

We at Meltwater have some custom plugins for Elasticsearch that make a number of modifications to how queries are executed, and completely replace some low level Elasticsearch query implementations. We're also running a custom ES 1.7 version with some features backported from version 2+. The end result is something like 5-10x lower GC pressure and radically increased performance and query throughput for our particular workload. Without these changes we'd not be able to sustain our workload without a massive amount of more hardware and cost, just like you say.

Our flavor of Elasticsearch 1.7 is faster than vanilla 2.* for our workload, though still slower than ES 2.* with our customizations applied.

Recent Elasticsearch versions still use the same basic shard allocation algorithms as far as we know. Our workload is very imbalanced towards recent data, but it's not a binary hot-cold matter, rather a more exponential decay in workload for older indexes. We fully expect to need Shardonnay to balance the workload even with ES 7+.

We're also in early conversations with Elastic about shard placement optimization. They seem to be interested in applying linear optimization in a similar way, with a goal of solving the fundamental problems with shard allocation based on observed workload.

In our case at Meltwater we also have new documents as well as updates to old documents coming in. Meaning older indexes are actively being modified with low latency requirements on visibility and consistency of those modifications. This makes it more tricky to both reindex as well as doing a seamless/no-downtime-whatsoever upgrade to a new major version. It's not infeasible at all though, and we're working on it. Reindexing a PB of data should be possible on a matter of weeks based on our estimates, but we shall see!

We ended up creating https://github.com/meltwater/secretary to allow storing encrypted secrets in config files checked into Git by devteams. The encrypted secrets are passed as env vars through the continuous delivery pipeline, Mesos/Marathon and into containers. They're then decrypted and injected into the app environment at runtime, safely inside the container.

At startup the container reaches out to the Secretary daemon that holds the master keys, using public key cryptos to authenticate itself. The Secretary deamon uses Marathon to authenticate containers (checking their public keys stored in env vars) and validate that they're authorized for the specific secret in question (checking that the encrypted secret is indeed part of the containers env vars).

Meaning that Marathon is the single source of trust of which container can access what secrets. The problem then becomes controlling who and how changes are made to the Git repo containing the CD config, which is something Github does well with roles, status/deployment API and pull requests.

We had a similar problem as some describe with the distribution of the initial secret (i.e. Vault token) and one time Vault tokens being cumbersome in dynamic scaling envs. We didn't want the cleartext token ending up in config files nor in the https://github.com/meltwater/lighter config we use to drive our continuous delivery pipelines that go into Mesos/Marathon. We also had some other aspects like

* Wanting to keep secrets, app config and code versions promoted together throughout or deployment pipelines. Seeing secrets as another type of app config we wanted to track all config and versions for an app in the same way, in the same place to avoid mismatches or deployment dependencies.

* Wanting to enable our very independent devteams to easily manage secrets for their services, same was as they manage the app config, versions and rollout of their services. And delegate management of what service is authorized for what secrets to devteams (with both automated checks for unencrypted secrets, and some gentle manual coaching post-commit)

* Versioning and rolling upgrades for secrets? E.g. how to roll out a new secret in a Marathon rolling upgrade? Creating and managing versioned keys in Vault seemed somewhat cumbersome.

Perhaps something like that could be used to solve your initial secret distribution problem or even handle the secrets themselves until Vault has solved the initial secret problem..?