Deploying Elasticsearch on a 150 node cluster to index 10B documents

Re:Deploying Elasticsearch on a 150 node cluster to index 10... by landryraccoon on Monday May 23, 2016 @08:17PM (#11758009)

> The only other way to determine the amount of documents returned it to JSON parse the entire result which is not free and would add latency to each request.

Why do they have to bill synchronously (in realtime)? Why not return the response, then calculate the fee asynchronously? The customer is probably being billed at discrete intervals (e.g., weekly or monthly) anyway.

[ Parent | on HN ]

Re:Deploying Elasticsearch on a 150 node cluster to index 10... by burtonator on Monday May 23, 2016 @08:29PM (#11758054)

It's a good idea (and thanks for the feedback) but we would still have to burn the CPU (which isn't free) to parse the response.

Much easier if it's just an integer header field.

[ Parent | on HN ]

Re:Deploying Elasticsearch on a 150 node cluster to index 10... by craigching on Monday May 23, 2016 @09:16PM (#11758307)

What's your request size and how big are the documents you're needing to parse? Even if you're returning 1000 documents, if you're not fetching the source as part of the request (do you need to?) or if you don't have a ton of fields, it shouldn't be too expensive to parse that to get the total number of hits.

[ Parent | on HN ]

Re:Deploying Elasticsearch on a 150 node cluster to index 10... by jonaf on Monday May 23, 2016 @09:42PM (#11758409)

I'm assuming he needs the source, otherwise he should look at the count API.

[ Parent | on HN ]

Re:Deploying Elasticsearch on a 150 node cluster to index 10... by craigching on Monday May 23, 2016 @10:18PM (#11758563)

The count API just returns counts. I'm assuming he needs some fields to display which should be smaller than source if source is "large." Unfortunately we are left to guess at his statements because not enough detail is provided.

[ Parent | on HN ]