IMO it's also worth considering the issue of synchronous vs asynchronous logging during the process.

Throughput gets better but latency's a tougher nut to crack.

> Given that the cost of logging is low and continuously falling

Everything gets cheaper in time, but the cost of logging is actually quite high.

I don't know why you're getting downvoted, because in my mind you're right in some circumstances. More than once, I've found logging-related performance issues (i.e. logging is near the top of a profiling-driven performance optimization task).

One immediate example that comes to mind was code that had a bunch of logging statements like log.debug(boost::format("%d %x etc", param, param)). Normally the code would run with debug logging turned off, but there's a pretty brutal problem there! The formatted string is always going to be processed, which the logger will immediately discard. Turns out one of these formatted strings actually took a fair bit of processing to generate and was hurting the system.

That's a solved problem. Your logger should be capable of formatting on it's own, and take the arguments to the format string as variadic arguments. This is how Java's ubiquitous SLF4J interface is defined. If the log level isn't low enough, then the arguments are tossed but never composited.

And it's even better with languages with macros, like Rust, Scala, even C++. There won't be any function call to begin with if the corresponding logging level is not enabled, only a branch.

I replaced it with a macro!

The cost of logging is actually quite low relative to the enormous cost of engineering hours spent debugging a system that's flying blind.

I don't know who downvoted you, though I still disagree.

If I was writing an embedded system, or a system with soft-to-hard realtime requirements, or one where flat straight line performance was the only requirement that counted, sure, I might ditch or constrain logging.

These systems might be common by manufactured volume. But most programmers will never touch one.

You're probably thinking of I/O costs. Again, these aren't as a brutal as they used to be -- SSDs swallow bursty writes with aplomb and when those aren't enough, log-structured filesystems are good at smoothing out writes.

As for myself, in the work I do in my day job, I print out to STDOUT/STDERR and let the platform wick those lines away to a central firehose.

   > If I was writing an embedded system, or a system 
   > with soft-to-hard realtime requirements, or one 
   > where flat straight line performance was the only 
   > requirement that counted, sure, I might ditch or
   > constrain logging.

I've worked on embedded systems, and logging is still extremely important, if not more important because of the difficulty of seeing what's going on (it's hard to reproduce events that only happen in a truck while you're inside your office).

In one case, a team had been trying for months to debug a problem, but when proper logging was added, the issue was easily fixed.

> in the work I do in my day job, I print out to STDOUT/STDERR and let the platform wick those lines away to a central firehose.

I think this is the correct approach, but (to the nearest approximation) ~every organization I've worked at that does this has hit saturation limits of "the platform" in extremely short order. Especially if you're doing audit-trail-style logging, in my experience, this becomes the primary bottleneck of your infrastructure very quickly.

Did you come up with particular tactics? Any insights on platform bottlenecks?

I am curious about this. Can you give specifics about the costs of logging in this context?

Not the parent, but I know our production web servers have very limited logging since they were already highly optimized and handle a huge amount of traffic, so logging was taking up something like 15% of all CPU time (and since they were CPU-bound, 15% of cost), so we have stats like those described in this article instead.