Re:Storing 50M events per second in Elasticsearch
by MuffinFlavored on Tuesday October 22, 2019 @10:19PM
(#21330447)
200k documents per second is a lot less impressive, no?
[ Parent
| on HN ]
Re:Storing 50M events per second in Elasticsearch
by virgilp on Wednesday October 23, 2019 @03:41AM
(#21332148)
I don't know. Have you tried it? I worked on a Kafka streams service written in Java that processed "changelog" messages (it involved one query to CosmosDB per message, and logging the result to Kafka for downstream processing by other systems). Now, we had a rather limited number of workers (4 or 8? I don't remmember), but getting to 100k messages per second was rather challenging.
Re:Storing 50M events per second in Elasticsearch
by hmschreck on Wednesday October 23, 2019 @10:54AM
(#21335230)
200k documents per second really isn't that much for Elasticsearch. The single-instance setup we have in my small company (around 25 people total) has been sent in excess of 40k/s at times, and even then it doesn't slow down noticeably.
Re:Storing 50M events per second in Elasticsearch
by Scarbutt on Tuesday October 22, 2019 @10:36PM
(#21330546)
and less catchy.
Re:Storing 50M events per second in Elasticsearch
by Benfromparis on Wednesday October 23, 2019 @01:53AM
(#21331491)
"200k documents per second" would be less catchy you are right, but it also doesn't reflect the complexity to handle that throughput.
Most of the benchmark we can see around like "1m document per second" are using small documents in POC environement.
In our setup, each of the 250 fields are store and indexed in ES, making it CPU and I/O intensive, in a production environment.