News for hackers. Stuff that matters.
Re:Storing 50M events per second in Elasticsearch by altmind on Tuesday October 22, 2019 @09:03PM (#21330076)
>> Each day, during peak charge, our Elasticsearch cluster writes more than 200 000 documents per second

What is this 50M in the title?

Re:Storing 50M events per second in Elasticsearch by gatorcode on Tuesday October 22, 2019 @09:17PM (#21330153)
They state each document has 250 events, 200,000 document/sec x 250 events/document = 50m events/sec.
Re:Storing 50M events per second in Elasticsearch by kodyo on Tuesday October 22, 2019 @09:52PM (#21330336)
"Each document stores 250 events in a seperate field."

Curiouser and curiouser.

Re:Storing 50M events per second in Elasticsearch by Benfromparis on Wednesday October 23, 2019 @01:47AM (#21331455)
Good catch! We will add it in the article
Re:Storing 50M events per second in Elasticsearch by uncoder0 on Tuesday October 22, 2019 @09:47PM (#21330310)
It's misleading for sure but they're writing 250 'events' per document.
Re:Storing 50M events per second in Elasticsearch by MuffinFlavored on Tuesday October 22, 2019 @10:19PM (#21330447)
200k documents per second is a lot less impressive, no?
Re:Storing 50M events per second in Elasticsearch by virgilp on Wednesday October 23, 2019 @03:41AM (#21332148)
I don't know. Have you tried it? I worked on a Kafka streams service written in Java that processed "changelog" messages (it involved one query to CosmosDB per message, and logging the result to Kafka for downstream processing by other systems). Now, we had a rather limited number of workers (4 or 8? I don't remmember), but getting to 100k messages per second was rather challenging.
Re:Storing 50M events per second in Elasticsearch by hmschreck on Wednesday October 23, 2019 @10:54AM (#21335230)
200k documents per second really isn't that much for Elasticsearch. The single-instance setup we have in my small company (around 25 people total) has been sent in excess of 40k/s at times, and even then it doesn't slow down noticeably.
Re:Storing 50M events per second in Elasticsearch by Scarbutt on Tuesday October 22, 2019 @10:36PM (#21330546)
and less catchy.
Re:Storing 50M events per second in Elasticsearch by Benfromparis on Wednesday October 23, 2019 @01:53AM (#21331491)
"200k documents per second" would be less catchy you are right, but it also doesn't reflect the complexity to handle that throughput. Most of the benchmark we can see around like "1m document per second" are using small documents in POC environement. In our setup, each of the 250 fields are store and indexed in ES, making it CPU and I/O intensive, in a production environment.