Twitter util to the rescue! https://github.com/twitter/util/blob/master/util-core/src/ma...

We actually built something quite similar to that, but how do you avoid having to artificially insert a save() and restore(context) call all over the place? It looks to me like you'd have to do something like val context = Local.save() val eventuallyFoo = someServiceClient.makeCall("data").map { result => Local.restore(context) //do work }

for every single async interaction.

That's what we do, but it's built into the RPC library we use, finagle: https://github.com/twitter/finagle/blob/master/finagle-core/...

BTW, to the readers at home, note how almost all our core infrastructure is open-sourced.

That's intriguing.

Could you please elaborate on that? Does this mean that Twitter sees no danger in exposing these? Which parts of the infrastructure would be considered secret sauce and not open source? Or does it not matter when the company is as big as Twitter, since the core strength lies in user base, not technical infrastructure? What does Twitter primarily seek to achieve when it open sources its stuff? Talent acquisition or brand image or other benefits of open source such as collaboration?

More or less 100% of asynchronous composition is done via Futures[1] whose default implementation[2] does this for you.

[1] https://github.com/twitter/util/blob/master/util-core/src/ma... [2] https://github.com/twitter/util/blob/master/util-core/src/ma...

I mean this as politely and constructively as possible, but this looks like it has quite a few easy to fall into failure modes. If I'm not using a twitter Future, or if I've been given a Future from somebody else, it would look like I need to ensure to surround every compositional operation with this save and restore, otherwise my context is permanently lost. Additionally, I have to trust that any code I hand a closure to will do the right thing, otherwise my context is potentially lost since I have no guarantee on which thread that closure will actually be executed on. It seems like it would be virtually impossible to avoid this happening at least once in even a simple codebase, and when context is lost, it happens silently.

You are right that this fails when you move outside of the model. However, we don't.

You may be surprised (amazed?) to learn that, internally, 100% of composition happens in this manner. We have a massive code base, and we've not seen this be an issue.

Further, we've worked with the Scala community to standardize the idea of an "execution context" which helps make these ideas portable, the particular of the implementation transparent to arbitrary producers and consumers of futures, so long as they comply to the standard Scala future API. (Twitter futures will when we migrate to Scala 2.10.)

This is true, and I look forward to the Typesafe guys fixing both the default implicit ExecutionContext as well as any contexts that Akka creates. I actually implemented an ExecutionContext that does exactly what is described above, but ultimately we had to abandon it, since pretty much any library that deals with scala standard Futures in 2.10 has places where we could not provide our custom context. I can't wait until you guys get that ported upstream, because until then, I've explicitly banned the use of thread locals across our entire stack.

Yes, if you "go off the reservation" and outside of the Twitter Future, you will lose your automatic trace identifiers.

This isn't a unique problem, but using consistent libraries goes a long way (which works well internally at Twitter).

(For fun, I just searched through our entire codebase. The only manipulation of locals anywhere is in the util library.)