Re:Distributed Logging Architecture in the Container Era
by ktamura on Thursday August 11, 2016 @11:56PM
(#12273314)
Ah, I completely misunderstood. Yes, you are right: Threading data together across multiple containers belonging to a single logical request is hard (and AFAIK, none of the container orchestration tools deal with it well).
Re:Distributed Logging Architecture in the Container Era
by bogomipz on Friday August 12, 2016 @12:04PM
(#12276699)
How is the unique trace ID implemented? Vector clocks would be in use by your distributed data store for conflict resolution such as Riak does. Could you elaborate on how you've implemented such a system?
Re:Distributed Logging Architecture in the Container Era
by rdli on Friday August 12, 2016 @12:47PM
(#12277024)
It's not about conflict resolution, it's about ordering.
Imagine you have 3 services. A calls B calls C. A request goes into A. The request fails. Since there are 3 services involved, you want to be able to see the log statements of all 3 services, in order. To do this, you need a UUID that is passed from A to B to C and a vector clock that is incremented from A to B to C so you get perfect ordering. Passing the data around is not super hard (you can use HTTP headers, for example). The tricky bit is having a library that does the right thing in A and B and C. And if A is written in Java and B is written in NodeJS, that library better work on both!
To solve this problem, you need a unique trace ID and vector clock, and this actually needs to propagated through your logging system, not through container tags.