News for hackers. Stuff that matters.
Re:Grafana Tempo, a scalable distributed tracing system by atombender on Thursday October 29, 2020 @11:59AM (#24931903)
We log probably from 30+ services, plus third party apps. Standardizing on a schema is to some extent possible (but hard to actually enforce and test; many of our services are written in JS/TS, not Go), but not for third-party apps.

Could one set up FileBeat or whatever to ingest into separate indexes based on some source label? Most of our logs are streamed from Kubernetes, so there is an application label we could use. Elasticsearch is pretty good at unioning a single query over multiple indexes.

But it'd have to be an automatic mapping — I wouldn't want the ops team to have to maintain a mapping between apps and indexes that could (would!) get out of sync.

Re:Grafana Tempo, a scalable distributed tracing system by axw on Friday October 30, 2020 @03:28AM (#24939932)
Yes, you can do this using Filebeat or in Elasticsearch itself using an Ingest node script processor.

Filebeat: https://www.elastic.co/guide/en/beats/filebeat/current/elast... (You can incorporate variables in the destination index; variables may refer to labels and other fields.)

Ingest node: https://www.elastic.co/guide/en/elasticsearch/reference/curr... (search for "_index")