stochastimus

This is all true. So for the times you end up there, wouldn’t you prefer a tool to surface for you the things you were going to have to spend hours digging for?

We go back and forth on this as well. When I talk to people I will usually say something like “we find the stuff in the logs you were going to have to dig for, and surface it in a report on your dashboard”. I think the RCaaS is in that sense aspirational, and there’s a bunch of other stuff you’d want to do as well like integrate log and metric AD for example. I really appreciate your thoughtful comment and I think you’re right, we can still communicate better about this concept with the broader community. Pls do reach out if you’d be willing, Id love to pick ur brain over a drink or w/e. [email protected]

I feel that vendors have really muddied the playing field by claiming ML features and then not delivering value. That’s why we tend not to talk about ML. But, I think saying that ML is unhelpful for the problem is a bit like saying programming is unhelpful for the problem - the solution space is vast, and has hardly been explored.

We do indeed since we need to process an amount of data in proportion to that volume, and so more resources are required - you can also run it on-prem, if you want

It's really quite inexpensive - but would love to get your feedback on pricing, esp. at scale! pls. do reach out

LoLz

Hey! Founder here. Thanks for your comment!

> That's not hard to debug

I agree there's a large class of problems where that's the case, and I also agree it's perhaps easier than many of the issues we've caught in the real world, such as are discussed in the Cisco case study. But I still want to defend the example as non-trivial.

So of course we pick up the errors in services that are fall-out from the breakage, and we pick up the chaos test running, which leaves a pretty wide footprint. But if something like this happened in the real world, the important event would have been that one that's in the RC report, that's not even an error (though it is quite unusual): the kernel log entry pointing out the eth0 misconfig. It can take a long time for someone to get around to poring through their various host kernel logs and looking at everything, even non-errors, so having it surfaced to you right away feels like a very useful thing.

At the same time though, I like the example you gave even more. What many people will do is upload their own incident data via the CLI, or deploy our chart into a staging environment and just break things, to see what happens. Based on your description, there's a good chance we would pick it up since we track anomalies and perform anomaly correlation across all pairs of streams: we're not looking for a threshold percentage of overall error rates, for example, anywhere. If you'd be willing to help us test this use-case, I'd love to work with you personally to help you get up-and-running.

Also, as we see more and more failure modes, we continue to make our detection algorithms more robust. While we generally achieve >>90% detection of incidents and their root cause indicators overall, there are always places we can and would love to do better. I think you'd find that our solution would catch most of what you'd want out-of-the-box, and that we're responsive enough to learn from every customer.

Lastly, I would ask: do you think that we should do videos of some more "subtle" examples (for lack of a better word)? As someone viewing the website, would you have watched them?

Thanks again for your feedback!

This won’t be exactly what you were looking for necessarily but when I read this I felt compelled to respond. Twice I have failed in my pursuits in such a way that going forward was essentially “starting over”; in both of these cases there were residual consequences such as debt. It was heartbreaking for me and I don’t know exactly what it feels like for you right now but I imagine it is a heartbreaking experience for you right now, as well. Let me tell you two things, please, based only on my experience.

First: in both cases things got better sooner than I had expected; within a few years things were good... once I had dusted myself off and applied myself to the next chapter of life, I was able to take steps, find ways to dig out, and find hope and passion again in my pursuits.

Second: I, like you, had a lot to offer the world, my profession, and my family and friends; I look back and can see these things. Please, hang in there, and please, don’t let the downturns extinguish your light. Whether you know it or not now, many people will need you around, and in the end, even more will be better off for your influence.

Awesome - my first thought was “hmm I wonder when the Julia community will implement something based on this” :)

Freeman was right after all.

does this mean that goto is not always considered harmful? ;-)

Nope, completely custom, with a big chunk written prior to Flux anyway, but I’ve seen Flux and I’ll be looking for opportunities to play with it in future projects for sure.

I enjoy seeing this sort of press for Julia. One thing I’ll say for the language is that it is good with elegant high level abstractions for productivity while at the same time being able to go as low-level as you need for optimization. My company uses Julia for our text structuring ML. From go we communicate with the Julia binary using protobufs. I have no complaints and have enjoyed my experience with Julia.

Kudos - I won’t remember a lot of the new products I’ve seen websites for, but I’ll remember this one! :)

The unmentioned benefits: it is fixed-width and follows most to least significant parts left-to-right. What this means is that the significance of the parts follows the same L-R convention as numbers, making it intuitively obvious which is the month and which is the day. Oh, and a simple string sort will sort timestamps correctly - very handy at the command line. Even db columns holding timestamps as strings will get the ordering right by default.

Here’s a question for you - for a typical L2 support person or front line MSP NOC monitor, or junior SRE, or similar - what do you think would be the error rate? I wonder if there isn’t sort of a front-end role here that can be subsumed at least in part by automatic summarization, for the purpose of streamlining the escalation and routing process overall, even though errors would occur - since I imagine they already do. What do you think?

Yeah, I'm really hoping there's a way to reliably corral it into basic plain-language restatement of the logs themselves without it going too far afield or speculating. We will see.

Great question. I guess for me as a technologist, I’m coming at it from the other side and asking what failure rate is possible, and can that rate be reduced with further prompt fiddling and other machinations? Then I will try to gauge whether that rate is acceptable to users. I guess since it’s all so new, people wouldn’t know what would be useful or usable until they tried it. So we are looking for users to participate and help us find if there’s a viable product involving GPT-3 summaries.

Yeah, that’s a fair way to look at it. Thanks for that. It also matches my experience with giving it too much unrelated input data: even the first line of the summary will often be completely fictitious in that case.

Thanks for putting that info here!

Well, I've been trying really hard not to point it out because I don't want this to be like a commercial. :) But, the idea here is that the Zebrium ML picks the incident lines unsupervised; then, the GPT-3 model creates the summary unsupervised. So I guess the combination is what we've been working on in a private beta, so that the user can get the best of both worlds.

From what I've read, the answer is "yes", stackoverflow was crawled. EDIT: I looked and stackoverflow is included in the Common Crawl dataset, which is one of the datasets on which GPT-3 was trained. Having said that, it's not clear to me the degree of coverage of that domain guaranteed by that crawl... looks pretty comprehensive, though. http://index.commoncrawl.org/CC-MAIN-2020-24-index?url=*.sta...

I just tried this and I might stick with these settings! ;-) For the postgresql example in the blog, I used your prompt. Here's what I got:

The logs were in a mess, But the expert could see, That the database was in distress.

Hmm, I like this direction - so maybe, as the user is navigating the incident, let them steer the model with questions and/or additional lines. Is that sort of what you'd envision?

Yeah, my experience so far has been that if I just pile a bunch of logs in there, if it's not given salient lines, the language model tends to either rat-hole on some particular detail that's irrelevant, or else construct a non-factual narrative. But, when Zebrium did the picking of these lines autonomously and GPT-3 summarized them, we see a meaningful summary. Having said that, I would also like to get GPT-3 more and more savvy with larger and larger log-based prompts, and I'm hoping it's possible with some tweaks to the prompt and some finetuning. I'll keep the blog posted as we do more experiments.