News for hackers. Stuff that matters.
Re:Building resilient services at Prime Video with chaos eng... by BiteCode_dev on Monday August 24, 2020 @01:34AM (#24257775)
Is this a case of "netflix did those articles, so we'll do the same so geeks like us too"?
Re:Building resilient services at Prime Video with chaos eng... by danellis on Monday August 24, 2020 @02:02AM (#24257891)
It does seem like a fad. Kudos to those who can create a new field and profit from it. On the one hand, "chaos engineering" seem a bit like "we don't understand our architecture well enough to know what its failure modes are, so let's just poke it and see what happens," but on the other hand, it seems at least a little bit analogous to fuzzing, which is certainly a technique that yields useful results that would have otherwise been overlooked until it was too late.
Re:Building resilient services at Prime Video with chaos eng... by mohave529 on Monday August 24, 2020 @02:33AM (#24258021)
My first instinct was to agree with this, but from my experience it's extremely difficult to properly communicate failure modes 100% of the time across different teams in very large organizations. Dependencies that are fuzzy arise for example when a service A proxies data for client service B from some other service C. It doesn't help that the organization of teams in a company often severs lines of communication between teams who explicitly don't have dependencies but implicitly do. As a result, information gets lost in the process. Having a last line of defense in the form of a "chaos engineering" team may actually be the natural response of large organizations to counter the inherent messiness that is produced as a result of bureaucracy.
Re:Building resilient services at Prime Video with chaos eng... by lhoff on Monday August 24, 2020 @05:28AM (#24258881)
That and additionally it has implications for the development team as well. Using "chaos engineering" shifts the mindset of the developers. As a developer you now expect things to fail. You know that the "we make it work first and make it resilient later" approach will bite you sooner then later so you think resilience from the first line of code.
Re:Building resilient services at Prime Video with chaos eng... by twblalock on Monday August 24, 2020 @03:12AM (#24258188)
You don't know what all your failure modes are. You probably think you do, but you don't. That's the point of chaos testing.
Re:Building resilient services at Prime Video with chaos eng... by kokooo on Monday August 24, 2020 @03:01AM (#24258138)
There are a million different ways a computer can fail. I think we're asking too much of people to be able to know all the pitfalls of every system they create.

But also this 'new field' just seems like something we've already been doing just with a different name. You're kind of expected to make sure your system can work if the computer suddenly shuts off, or a dependency is lost, or the network is slow. Have we not been doing this??

Re:Building resilient services at Prime Video with chaos eng... by danellis on Wednesday August 26, 2020 @05:43AM (#24280829)
It does seem antithetical to engineering, though. Are there other engineering fields that take this approach?
Re:Building resilient services at Prime Video with chaos eng... by eru on Monday August 24, 2020 @02:29AM (#24258007)
> On the one hand, "chaos engineering" seem a bit like "we don't understand our architecture well enough to know what its failure modes are, [...]

That approach seems like a good idea, even if you think you know what the failure modes are.

Re:Building resilient services at Prime Video with chaos eng... by scarmig on Monday August 24, 2020 @01:42AM (#24257807)
If nothing else, the article showcases how to use the recently released AWSSSMChaosRunner[0], which I hadn't heard about. Beyond that, it's good when multiple different high quality blog posts show how to do what's fundamentally the same thing.

[0] https://github.com/amzn/awsssmchaosrunner

Re:Building resilient services at Prime Video with chaos eng... by marmshallow on Monday August 24, 2020 @02:20AM (#24257957)
Definitely seems like a case of someone that wanted to publish a blog post for some internal kudos, but I'm not complaining. Always good to share knowledge.
Re:Building resilient services at Prime Video with chaos eng... by smabie on Monday August 24, 2020 @02:40AM (#24258045)
It sounds like complaining to me
Re:Building resilient services at Prime Video with chaos eng... by cheeze on Monday August 24, 2020 @02:10AM (#24257920)
My guess? Someones manager was like "Hey lets publish this wiki as a blog post."

Sure, part of it is trying to seem cool and whatnot, but that's fine with me. The output is the same, more knowledge shared.

Re:Building resilient services at Prime Video with chaos eng... by jwilber on Monday August 24, 2020 @02:31AM (#24258017)
Never thought I’d see anti-article sentiment on HN, and more generally, not sure what sort of response you expect to generate. Why comment at all?

To avoid more of the same, I’ll say that the article itself is pretty detailed, but whereas Netflix blog-posts are more general, this one feels hyper-specific. That’s great if you’re tied to aws, but I think the former has longer-lasting + further-reaching utility.

Hope they do more.

Re:Building resilient services at Prime Video with chaos eng... by BiteCode_dev on Monday August 24, 2020 @04:47AM (#24258645)
Not an anti-article sentiment, but I dislike the trend of big companies pretending they care about the nerds just because they realize they need them for one particular objective.

It's an anti-marketing sentiment.

Re:Building resilient services at Prime Video with chaos eng... by jwilber on Monday August 24, 2020 @01:53PM (#24263320)
That makes more sense. I think it’s worth framing differently, though.

Big companies aren’t writing the doc, per se. Engineers interested in the material are. And often, they want the opportunity to do so - it’s something of a stamp for “expertise” in their favor.

Assuming the post is actual quality, it’s a marketing win for the company, a professional/developmental win for the employee, and an informational win for everyone else.