There's a huge operational and engineering cost buried in that statement, which the article is trying to unearth and provide better solutions for.
> 5. Logs should also be distributed, in the sense that you don't ship all your logs to a central location, but rather have each server manage its own logs.
Though it does give you some nice properties, this is counter to a lot of accepted best practice... is there a good distributed log query system? `dsh` doesn't cut it, for the record :)
Sure, there's always an operational and engineering cost to logging. I think the original comment makes that plain.
Perhaps there was an unstated assumption on your part, but how are my app servers to know which one handles which query? Is there some sort of log coherence system going on? Are you running Hadoop on your app servers?
You keep on using that word... but seriously, thanks for the non-answer, now I can be confident you don't know what you're talking about.
I guess your original comment made clear that you weren't really interested in an answer beyond dealing with "unstated assumption" and that you want to persist in thinking of logging from a centralized point of view. You just wanted to be "confident".
But if you're seriously interested in discussing further to see what I'm working on, feel free to drop me an email on [email protected] and we can setup a Skype face-to-face.
2. You almost want your log entries to tell a story, you want to spend nearly as much time on your log entries as on your code. I think the more you get this right, the better your software design will be.
3. You also want to be able to grep your logs easily using a transaction and user ID. For example, if you just fetched a batch of emails over IMAP, you want to have a transaction ID to describe the session, and include this in all related log entries for the session. You also want to include an account ID in all related entries so you can see all fetches for a given user.
4. Your log entries should do as much work for you as possible. For example, if your log entry logs a filename or string that contains unusual Unicode characters, then you might want your log function to also log what Unicode normalization form the string contains (NFC, NFD, Mixed). This can make otherwise complex bugs trivial to solve.
5. Logs should also be distributed, in the sense that you don't ship all your logs to a central location, but rather have each server manage its own logs. If you need to query your logs, then send a query out to the relevant servers and get them to do the work for you. If logs are always a fraction of a server's data output then you can easily manage gigabytes of log data per server this way.