Ajs1
Ajs1 on Hacker News
Posted by
Ajs1 on Monday March 14, 2022 @04:08PM
from the self-post dept.
https://www.zebrium.com/blog/how-cisco-uses-zebrium-ml-to-analyze-logs-for-root-causeDisclosure: I’m one of the co-founders behind the ML mentioned here. However the results in the link above were generated independently by Cisco.There have been a number of write-ups and papers evaluating ML for logs over the last few years – by academics, engineers, and observability vendors. But these results...
Posted by
Ajs1 on Thursday April 15, 2021 @01:37PM
from the zebrium.com dept.
appreciate your comment - spot on!
fair point, but this blog and your comment is about a summary sentence. if you read through to how the underlying log reports are constructed, those are very accurate (and also quite concise).
Right - we will consume alerts from other tools. I see your point about consolidating collection. We'll look for opportunities like this if we can. For now, we do try to make it easy and lightweight to set up our collectors. And many of our users do have multiple collectors/agents on the same clusters. Please contact us if you'd like to give it a try.
Hi paridso, we tackle the problem at a more foundational level.
AIops tools are designed to speed up resolution and reduce noise, but they act on a feed of alerts/incidents. So in a sense they are dependant on the quality of alerts generated by monitoring tools. And typically a human will end up drilling down into data like logs and metrics to determine root cause.
Our ML acts on the raw data, and has better coverage than typical log/metrics monitoring tools (including previously unknown failure modes). It also cuts time to root cause by generating complete incident summaries.
hi gingerlime, thank you. One thing we're working on is taking an incident signal from other tools and augmenting it with an incident report. We're starting with PagerDuty and Slack integrations as sources, but could see extending that to other APM/monitoring tools (DataDog could fit here). Of course we do have some overlap in the latter case (for logs & metrics). If you have something more specific in mind - let's connect.
There's more detail on the above mentioned plan here:
https://www.zebrium.com/blog/youve-nailed-incident-detection...
:)
founder here:
well, if there is a runbook for a known failure, we can trigger it via webhook. But auto-remediation for a previously unknown failure is of course a much different beast. Ambition for the future...
founder here:
you certainly can. Our goal is to minimize this need for you, but any team with experience already has some problem signatures/alerts for known issues, and we've tried to make it easy to capture those.
Our ML helps even this chore in one way - if you're building a signature relying on a log event - normally this is done with regexes, but you're at the mercy of a developer not changing syntax. Our ML will track these and ensure they signatures don't break if the log format changes in a future rev.
csears, founder here - could not agree more with your comments.
1. we learnt early to make user feedback easy (and immediately actionable) - they can quickly "like", "mute" and "spam". Or go more granular if needed.
2. This is an insightful comment. A few of our early users gave us similar feedback, and we've been hard at work. We'll soon be releasing a mode that takes an incident signal from your incident management tool such as PagerDuty or even Slack (often people create a Slack workspace per incident), and constructs a report around it.
3&4 are good points as well. Don't disagree about enrichment, just need to stage things.
Hi, one of the founders here: The service is designed to not require specific data models, because that does not scale, nor does it keep up with changes in application behavior.
Instead, the ML engine learns data structures, normal behavior of logs and metrics, and normal correlations between them for each app deployment on the fly. Then when things break it does a very good job of generating incidents. We make user feedback easy, so if we are "over-eager" in detecting a certain kind of incident, your response trains the ML quickly.
We do improve the ML engine with experience of course (and have added some user controls), but now have dozens of applications using us, and cumulatively have over a thousand successfully detected incidents under our belts.
and although this isn't quite answering your question about unknown/unknowns, this open source project lets you inject failure modes using a chaos tool (litmus) on your own app. We had really good results catching application incidents created by these chaos test.
https://github.com/zebrium/zebrium-kubernetes-demo
thanks Samdung. Appreciate it, and any feedback once you try it.
Posted by
Ajs1 on Wednesday May 20, 2020 @01:11PM
from the zebrium.com dept.
Posted by
Ajs1 on Tuesday February 25, 2020 @05:22PM
from the zebrium.com dept.
Posted by
Ajs1 on Saturday November 02, 2019 @08:02PM
from the zebrium.com dept.
Posted by
Ajs1 on Sunday October 06, 2019 @08:10PM
from the researchgate.net dept.
Co-founder here.
Understand your concern and perspective Brad.
In the near term this is a SaaS offering, so the best we can do to address these concerns is to offer strong security controls (a short list on our site, longer list on request) backed up by external certification (underway).
Plus the credibility and track records of the team behind it.
Posted by
Ajs1 on Thursday August 22, 2019 @10:06AM
from the zebrium.com dept.
Posted by
Ajs1 on Monday July 29, 2019 @12:14PM
from the geek.com dept.
Posted by
Ajs1 on Sunday July 28, 2019 @10:20PM
from the techcrunch.com dept.
no disagreement, I subscribe to 3 of the 4.
regardless of how you view the criminal justice system, the nationwide backlog of rape kits feels unconscionable. These are not quality of life or minor drug offences, so I'm surprised there isn't more public outrage.
Posted by
Ajs1 on Sunday July 14, 2019 @07:35PM
from the zebrium.com dept.
Surprising stats. I’ve been aware of mech Turk a long time, but did not realize average pay per task is quite that low
Nice summary of the (suboptimal) tradeoffs. Perhaps AWS is waiting for some customer push to offer better options.