News for hackers. Stuff that matters.
Re:Jaeger – A Distributed Tracing System by necubi on Sunday September 24, 2017 @04:07PM (#15326272)
I'm disappointed that all of the open-source tracing systems have adopted the Dapper [0] model. It's understandable why: it's extremely simple to implement, as it handles scaling challenges by doing client-side sampling.

A bit of background about how Dapper-style distributed tracing works. Things typically start with an RPC call of some kind (typically from an external source like a public load balancer). At that point, you must decide whether to trace this request or not, which is typically done as a random sample (say, 1% of requests). At that point, the request gets assigned a _trace id_, a random identifier for that request.

The trace id is stored in some request context and propagated to each subsequent service. Each service, meanwhile, divides up its request processing flow into a series of "spans" which represent some piece of computation. For example, a span cover an RPC call or a DB query. Spans are identified by a random _span id_. Once a request has been sampled, all spans for that request are sent to a central span collector where they're stored for later querying.

This model is simple but very limited. It's often hard to know whether a trace is interesting at the outset, hence the reliance on random sampling. For example, you might want to understand why your 99p latency is high, but if you're just sampling 1% the 99p requests will only be 0.01% of your sample.

More generally, interesting events (like errors or slow requests) tend to be rare, and sampling a random, small percent of requests is unlikely to turn up the interesting cases.

A better model, as implemented by lightstep [1] (and an in-progress distributed tracer I've been working on) is to collect all spans. Even with very high request volume it's reasonable to store all spans for at least a few minutes. Doing so opens up all sorts of interesting possibilities, because you can start tracing a request at any point during that window. For example, you might want to trace all requests that have errors in them. Or all requests that take longer than a certain time. Or get a google sample of requests across different latency buckets. Or requests that violate some application invariants you've defined.

Ultimately, though, distributed tracing is so helpful for understanding complex distributed systems and webs of microservices, and it's exciting to see more open-source competition for zipkin.

[0] Dapper is Google's distributed tracing system. The paper (https://research.google.com/pubs/pub36356.html) kicked off a lot of interest in distributed tracing in the broader community. [1] http://lightstep.com/

Re:Jaeger – A Distributed Tracing System by the_evacuator on Sunday September 24, 2017 @04:16PM (#15326312)
That sounds pretty expensive. With dapper the trace span annotations do nothing if the request isn’t being traced. If it is being traced you might have significant costs, along the lines of sprintf(“%.03f”, ...) or other very cpu-intensive activity. This is OK when you trace one per million but when you trace everything you have to think about the cost. This could lead to either just using more CPU than you really wanted, or to discourage trace annotations. Either would be bad.
Re:Jaeger – A Distributed Tracing System by necubi on Sunday September 24, 2017 @04:35PM (#15326377)
Yeah, that's definitely an upside of the dapper approach--very, very minimal overhead for non-traced requests. However for the vast majority of use cases a bit of overhead (microseconds per span) tends to be unnoticeable, and the benefits in terms of introspectability are huge. In general, the overhead is mitigated by the fact that spans tend to be pretty large (on the ms-scale).
Re:Jaeger – A Distributed Tracing System by ehsankia on Sunday September 24, 2017 @07:28PM (#15327064)
I guess it depends on your use case. If you are indeed interested in the 99p latency, then yes, the only way would be to trace everything. But in that case, couldn't you temporarily set the Dapper probability to 100%, record the data you need, and then turn it back down. It seems like a lot more malleable for different use cases.
Re:Jaeger – A Distributed Tracing System by foota on Sunday September 24, 2017 @04:32PM (#15326368)
Seems like you could store the raw arguments to printf and then only process them later if you decide you want the trace?
Re:Jaeger – A Distributed Tracing System by sulam on Sunday September 24, 2017 @11:45PM (#15327933)
In order to store them you have to serialize them.
Re:Jaeger – A Distributed Tracing System by justinclift on Sunday September 24, 2017 @04:20PM (#15326325)
LightStep sounds interesting, but it seems to be closed source?

That being the case, it's not hard to see why people are going with the (existing) OSS solutions. :/

UPDATE - Found this on GitHub, is this the whole thing?

https://github.com/lightstep/lightstep-tracer-go

If so, pointing people towards it from the .com website might help get people trying it out, as the .com website makes it seem non-OSS. :)

Re:Jaeger – A Distributed Tracing System by necubi on Sunday September 24, 2017 @04:33PM (#15326370)
Lightstep is closed source. They have opensource libraries that get data into the system (like the go one you linked to) but to actually use it you need the hosted service.

I hope to be releasing an open-source version of that approach in the next several months.

Re:Jaeger – A Distributed Tracing System by javajumbo on Sunday September 24, 2017 @07:16PM (#15327032)
@necubi I do not disagree with you about the value of tail-based sampling. It has been an item on Jaeger roadmap from day one, but never of high enough priority to implement it. We're planning to do a POC by the end of this year. Jaeger's agent-based architecture was specifically designed to support that use case. If you're interested in collaborating on that, it would be great (https://github.com/uber/jaeger/issues/425).

Having said that, Dapper-style tracing systems with head-based sampling still provide enormous benefits that are often underestimated. In fact, even if we support tail-based sampling in the future, we're still going to run a certain portion of the traffic through probabilistic, head-based sampling, because it allows to reason about statistical patterns observed in the systems at large scale.

Re:Jaeger – A Distributed Tracing System by justinclift on Sunday September 24, 2017 @05:00PM (#15326475)
Thanks, please do. :)
Re:Jaeger – A Distributed Tracing System by pritianka on Sunday September 24, 2017 @07:32PM (#15327079)
100% agree with what necubi said. I work at LightStep and it's great to see OSS implementations coming out. The creators of LightStep actually are also the creators of Dapper haha.

LightStep specifically is meant for large-scale enterprise deployments and their specific needs and has focused on that for now.

Re:Jaeger – A Distributed Tracing System by jpkrohling_jpk on Monday September 25, 2017 @09:57AM (#15330484)
This might be interesting to you:

https://github.com/jaegertracing/jaeger/issues/425

The Dapper model is interesting, but Jaeger is not 100% based on that.