News for hackers. Stuff that matters.
Re:Think Prometheus, but for logs (not metrics). Simple, eff... by jedberg on Wednesday September 14, 2022 @12:34PM (#32839589)
When a customer reports a problem the first thing I try is to reproduce it (with logging on). If I can't reproduce it, does it matter? I can leave logging on and let them know I can't reproduce their problem but if it happens again to let me know.
Re:Think Prometheus, but for logs (not metrics). Simple, eff... by encoderer on Wednesday September 14, 2022 @12:36PM (#32839619)
In your background of consumer engineering, yes it probably does not matter. But it’s not the same in SaaS
Re:Think Prometheus, but for logs (not metrics). Simple, eff... by jedberg on Wednesday September 14, 2022 @12:42PM (#32839699)
I've worked in SaaS for years, and I don't have any examples in my history where that's the case. Do you have an example of a case where historical logging was the only way to track down a problem?
Re:Think Prometheus, but for logs (not metrics). Simple, eff... by encoderer on Wednesday September 14, 2022 @01:06PM (#32840066)
Yesterday I used logs to help a customer who had mis-configured their SAML SSO (the problem was on the IdP side). I use our database audit trail logs to understand the sequence of changes a customer makes that sometimes results in behavior that is surprising if you only look at the current state.

We put a big emphasis on the second S in SaaS and we want a magical 1-email support experience whenever possible. Audit logs help this.

Re:Think Prometheus, but for logs (not metrics). Simple, eff... by staticassertion on Wednesday September 14, 2022 @01:00PM (#32839980)
When do you turn the logs back off?
Re:Think Prometheus, but for logs (not metrics). Simple, eff... by jedberg on Wednesday September 14, 2022 @01:05PM (#32840043)
When the problem is verified solved.
Re:Think Prometheus, but for logs (not metrics). Simple, eff... by staticassertion on Wednesday September 14, 2022 @01:05PM (#32840059)
I can't imagine the logs are off very much of the time then
Re:Think Prometheus, but for logs (not metrics). Simple, eff... by jedberg on Wednesday September 14, 2022 @01:15PM (#32840185)
Well when you have microservices, hopefully only one or two of them need logging at a given time.