We put a big emphasis on the second S in SaaS and we want a magical 1-email support experience whenever possible. Audit logs help this.
Depends really how long your customers can wait.
They're particularly valuable for production issues. If something has gone wrong and alerts are pinging around, I appreciate the chance to figure out what it was from what the logs are telling me, rather than having to turn the logs on and try to trigger the same problem again deliberately in a production environment!
Edit: just realised who I was replying to. Netflix and Reddit might be very different environments with very different reliability requirements to a lot of other types of software.
As I mentioned in my post, that's the one case where I find value in temporary turning on logging, but having old logs around is awfully costly for the off chance that they help.
> tracing individual resources/requests through a bunch of microservices to better understand some unique situation or behaviour,
That's request tracing, which is different. Just storing a request trace is useful, but that's not what I mean by logging. I'm talking about server application logs.
> identifying whether a particular code path is actually in use and if so how heavily
Again that would be request tracing, not application logging.
> They're particularly valuable for production issues. If something has gone wrong and alerts are pinging around, I appreciate the chance to figure out what it was from what the logs are telling me, rather than having to turn the logs on and try to trigger the same problem again deliberately in a production environment!
If the problem doesn't happen again after you turn on logging, then the incident is over isn't it? The customer is no longer affected. If they are affected, then your logs will keep showing it.
> Having old logs around is awfully costly for the off chance that they help.
Depends massively on the log volume. I've worked at places that generated terrabytes of logs over the course of a few days, and required very large ES clusters to try to keep on top of it. There was definitely value in having those logs, but it only made sense to retain them for a couple of weeks. It's much less costly for low volume services.
> ...request tracing, which is different...
Ye-es, until it's not. If your current setup isn't yet developed or grown enough to have full request tracing, you can get a long way filtering aggregated logs by a correlation ID. I know it's no substitute for a fully feature tracing system, but in situations where you have limited resourcing you need to prioritise it can get you a long way
> If the problem doesn't happen again after you turn on logging, then the incident is over isn't it? The customer is no longer affected.
That massively depends on the impact of the incident, I think. If it's a "There was a problem playing this title, please try again" kind of deal, it's no biggie.
If it's a "my broadcast TV channel is off air" or "my financial system has crashed in an unexpected way trying to reconcile this transaction" type situation, you might be a little more worried - and the customer is likely going to be very upset for quite a while, and some people with very serious faces might come to talk to you about very large fines.
from what ive seen people who rely on logs do not have proper request tracing and/or metrics. its much simpler to dump a bunch of unformatted junk to stdout and throw elasticsearch at it, but not a good experience.
i agree overall, i find logs to be a last resort with really bad signal to noise ratio.
More mundanely, if I get paged about an error rate, the first thing I'm going to do is look at the logs to see what the errors are.
Our logs tell me the state of the application when the error occurred without having to reproduce it. They also tell me how many customers are having the issue, so we can set the severity of the incident at the appropriate level and reach out to those customers proactively if needed.
Sure, we might be able to do this with metrics or tracing instead, but that's often just duplicating information we're already logging. For example, by adding a correlation ID in my log metadata, I suddenly have end-to-end request tracing. It's not as pretty as what New Relic can produce, but it definitely works.
We still did rely on logs to troubleshoot and understand some issues.
- [0]: https://sentry.io
The only use I can think of for historical logs is security auditing. In which case there is only a subset of logs you really need to trace a breach and they don't need to be searchable online -- you can pull them back from cold storage if you need to check for an incident.
Moving and storing logs and making them all instantly searchable is awfully expensive for little benefit.
Edit2: Disclaimer: This opinion does not apply to government regulated services like financial services.