When a customer reports a problem the first thing I try is to reproduce it (with logging on). If I can't reproduce it, does it matter? I can leave logging on and let them know I can't reproduce their problem but if it happens again to let me know.

In your background of consumer engineering, yes it probably does not matter. But it’s not the same in SaaS

I've worked in SaaS for years, and I don't have any examples in my history where that's the case. Do you have an example of a case where historical logging was the only way to track down a problem?

Yesterday I used logs to help a customer who had mis-configured their SAML SSO (the problem was on the IdP side). I use our database audit trail logs to understand the sequence of changes a customer makes that sometimes results in behavior that is surprising if you only look at the current state.

We put a big emphasis on the second S in SaaS and we want a magical 1-email support experience whenever possible. Audit logs help this.

When do you turn the logs back off?

When the problem is verified solved.

I can't imagine the logs are off very much of the time then

Well when you have microservices, hopefully only one or two of them need logging at a given time.

Exactly. Also sometimes you discover some issue and you want to see if this is never happened before, sometimes, often?

Depends really how long your customers can wait.

Logging is different than metrics. Metrics should tell you if a problem has happened in the past.

You may have to. Debug logging can be very expensive to simply leave on in production.

You can also prune logs based on level at different time intervals.