I've made some changes/clarifications thanks to heads up from two of the Elasticsearch guys.

Thanks guys! :)

"Your lucene index must be in memory"… holy crap!

This isn't true, just fyi. Lucene actually relies on the OS' filesystem cache heavily. A Lucene index (e.g. an Elasticsearch shard) is composed of many small "segments". Each segment is an immutable file, which allows the OS to aggressively cache them.

But if your index is larger than available memory (very common), the OS will naturally start paging the segments on/off disk as required.

That's the root of the "give no more than 50% of your memory to ES" advice, since the other 50% is needed by the OS FS cache for Lucene segments

Source: I'm an Elasticsearch dev :)

Thanks for the heads up, not sure where I've picked up that idea - will update accordingly :)