Re:Grafana Tempo, a scalable distributed tracing system
by axw on Friday October 30, 2020 @03:28AM
(#24939932)
Yes, you can do this using Filebeat or in Elasticsearch itself using an Ingest node script processor.
Filebeat: https://www.elastic.co/guide/en/beats/filebeat/current/elast... (You can incorporate variables in the destination index; variables may refer to labels and other fields.)
Ingest node: https://www.elastic.co/guide/en/elasticsearch/reference/curr... (search for "_index")
Could one set up FileBeat or whatever to ingest into separate indexes based on some source label? Most of our logs are streamed from Kubernetes, so there is an application label we could use. Elasticsearch is pretty good at unioning a single query over multiple indexes.
But it'd have to be an automatic mapping — I wouldn't want the ops team to have to maintain a mapping between apps and indexes that could (would!) get out of sync.