News for hackers. Stuff that matters.
Re:Grafana Tempo, a scalable distributed tracing system by SEJeff on Wednesday October 28, 2020 @11:45AM (#24920123)
I'd love to see how this compares to Jaeger.
Re:Grafana Tempo, a scalable distributed tracing system by number101010 on Wednesday October 28, 2020 @12:38PM (#24920860)
The headline would be:

Jaeger supports native search, but requires Elastic or Cassandra.

Tempo relies on discovery from logs/exemplars, but puts everything in object storage (s3/gcs).

Tempo is cheaper and easier to operate but lacks native search.

Re:Grafana Tempo, a scalable distributed tracing system by strofcon on Wednesday October 28, 2020 @11:56AM (#24920280)
I'm hopeful that with it effectively being a KV store pointed at a "simple" object store (S3, GCS, etc), it will be dramatically simpler to manage than Jaeger, and much more performant.

Backing Jaeger with Elasticsearch or Cassandra was a nightmare. :-|

Re:Grafana Tempo, a scalable distributed tracing system by gouthamve on Wednesday October 28, 2020 @12:19PM (#24920600)
See some more details around the motivations and history here: https://gouthamve.dev/tempo-a-game-of-trade-offs/

It will be easier and cheaper to manage compared to Jaeger but doesn't yet have any ability to search inbuilt (which is one of the reasons Jaeger is expensive).

Re:Grafana Tempo, a scalable distributed tracing system by netingle on Wednesday October 28, 2020 @12:19PM (#24920602)
That’s our motivation and experience over the last 6 months running it internally at Grafana Labs. Much cheaper and easier to operate.

Not to bash Jaeger though, its more powerful than tempo in that it allows you to search for traces. Tempo is about integrating with Grafana, Loki and Prometheus for finding traces.

Re:Grafana Tempo, a scalable distributed tracing system by SEJeff on Wednesday October 28, 2020 @08:10PM (#24925661)
So if you're not running Loki, you're not going to be able to find traces using tempo? We use grafana and prom, but not loki.
Re:Grafana Tempo, a scalable distributed tracing system by annanay on Thursday October 29, 2020 @07:10AM (#24929222)
Since Tempo is a k/v store that can retrieve traces given a traceID, we need either a metric system that can store traceIDs in exemplars OR any logging framework to log traceIDs that can be copied over to the Tempo Query UI.
Re:Grafana Tempo, a scalable distributed tracing system by candiddevmike on Wednesday October 28, 2020 @12:39PM (#24920878)
Jaeger has a local storage option (badger) that is OK for "ephemeral" tracing. I use it for development environments and in production (where each app can have it's own Jaeger instance). The data gets wiped periodically, prometheus metrics are used for trending.
Re:Grafana Tempo, a scalable distributed tracing system by retzkek on Wednesday October 28, 2020 @12:57PM (#24921139)
> Backing Jaeger with Elasticsearch or Cassandra was a nightmare.

Unless you're already running one of these, in which case deploying Jaeger is very easy (at least, that was my experience with our Elasticsearch backend).

> much more performant.

You expect an object store to be more performant than Elasticsearch?