Yeah, Loggly is very solid. I've set up logging infrastructure for 2 companies using it now. Their structured logging parsing, searching, and alerting is fantastic.

Now that I understand the parent comment, I don't think this quite solves the problem he's asking unless all of your services are perfectly synchronized. No?

For us, messaging between services go through queues and a client request might start in a particular service and wind it's way through a few others to finish the work. You wouldn't have the same client requesting things from two different services simultaneously, though in fairly short order they come through. Our use case might be just simple enough that this wouldn't work for a lot of people.

Also, if I'm looking at access logs I'm not going to have that ID unless I start putting it in request headers. Right now the application logs and other monitoring tools pick it up though.

Edit: Didn't see mention of the clock on my first post, but it still stands for us that we're not in the millisecond timings between requests going in and out of services so we haven't had a problem with the timestamps from the systems they're on. Requests generally take a few seconds to process (handle a file upload of decent size, ETL some data, pump out some output document, etc) so each step through a service buys enough time before moving on to the next thing that we're not getting out of order log statements across each service.

> Edit: Didn't see mention of the clock on my first post, but it still stands for us that we're not in the millisecond timings

Vector clocks don't necessarily have much to do with clocks. It's an algorithm for generating order in a distributed system.

https://en.wikipedia.org/wiki/Vector_clock

Any tips for a beginner on optimizing usage of the dashboard? Not using it for much right now, and I feel like the interface is going to take awhile to optimize/learn.

How are you passing that UUID along the request chain? For instance, when one microservice calls another, does it have to be manually passed in?

My web stack checks for a specific header at each potential level of entry, and in the absence of a value, generates one. Then each layer's logger will log it back out. It's pretty straightforward for web requests - just use the plumbing that's already there, headers.

Some platforms will do this for you -- Cloud Foundry will show `x-cf-requestid` for anything that passed through the gorouter and back.