delias_

Outside of software, resilience engineering is an established field using this definition and disambiguating the others. Some info on the origins going back to the 70s http://erikhollnagel.com/ideas/resilience-engineering.html. It’s only the last 5-10 years that people in software have been getting involved

We haven’t acknowledged that trying to ‘find causes’ at all is misleading!

Causes are something we create and construct afterwards. They aren’t a primitive that make up incidents. They don’t fundamentally exist to be found.

Causal explanations limit what we find and learn and the irony is that root cause analysis is built on an idea that incidents can be fully comprehended—they can’t!

Instead think about the conditions that allowed an incident to occur. Separate out everything you think looks like a cause and explore each of them.

Talk about the properties and attributes that were present. There are so many aspects to incidents that aren’t even causes at all, and they don’t follow a linear chain of this-led-to-that.

People are presented with RCA and 5 whys as really getting to the bottom of what matters, but the reality is this approach is a linear simplification. We need to kick it up a notch and practice more holistic investigations of incidents.

Stop getting at why people didn’t do what they thought they should have done, and start getting to the point of what actually happened and how those actions seemed reasonable at the time.

https://www.oreilly.com/ideas/the-infinite-hows

It sounds like you might be interested in rhizomatic actor networks https://en.m.wikipedia.org/wiki/Actor–network_theory

Spring is probably the only thing. All the rest of the stuff on the tech blog and Netflix OSS is heavily AWS centric, even the container platform Titus.

Spring Cloud includes a lot of the Netflix OSS stuff though like Zuul and Archaius.

Running apache in front of tomcat is still a very common pattern. Mostly due to connection handling and configuration around it.

Aside from the not-Uber aspect, istio and envoy have driven a lot of excitement around lyft from a tech perspective. I wonder how much that factors in here or if I'm overestimating the hype.

I have an adjustable desk. It's really nice to be able to move around throughout the day, but for any thoughtful task that requires some amount focus, I have to sit. I'd like to read further collectively inconclusive research about that.

naive tip: give your app that is relying on github a SKIPGIT environment variable to ignore the bits around git operations, and just make local changes as an interim hack.

Having worked at an MMO dev, I think this talk's praises and criticisms are all super on point. The social stereotypes of tech are I think even more pronounced in the games industry, and it would be super if everyone were this conscientious and their companies empowered all employees to have a voice about this kind of thing.

The pitfalls can so easily slip through into emerging social media forms. Making aware the lessons learned from the pioneering days of social videogames is definitely not an easy thing to do simply due to the elitism and proprietary nature prevalent in the games industry.

>Etsy’s insistence on running its own servers rather than using cloud-based services and software offered by companies such as Google and Amazon—an emphasis that was known, under Dickerson, as “code as craft”

So many times people fail to look at problems in terms of trade-offs instead of right/wrong or fashionable/faus pax

Running your own infra vs public cloud has so many considerations beyond a simple price tag. Massive scale companies on AWS have hadoop clusters just to work out their goddamn bill. But there are considerations you have to make about the number of people to maintain something and how that affects your org on just a cultural/sustainability level, what the velocity of SaaS allows you, etc.

Running your own infra doesn't have to be full of cruft. And maybe the rolling-your-own approach to certain things is paradigmatic to the way your business works, maybe it isn't. You have to figure out the trade-offs.

Newer, fashionable tech is often decorated with tag lines of "faster, cheaper, better". That's actually not what I care about. Instead, sell me on "recoverable", "debuggable", and "sustainable".

Good feedback from peers I think is critical, but the 360 review process most companies have does not provide introspective and effective feedback.

Your bullet points can be addressed in productive 1-on-1s with your direct report manager and regular check ins between your team and project management to critically discuss recent milestones and what emergent work has been created from them.

I agree with your points, but I feel the formal corporate review process does not sufficiently facilitate them and instead succumb to the pitfalls highlighted where they only focus on the negatives. It is not that I only think reviews are for that, it's that I think those things are a bad end state where formal reviews converge.

The 360 review does not contribute meaningfully toward creating a learning organization. It is not the right mechanism for feedback and reparative methodologies.

We're not building technology. We are building organizations who are building technology. If we are not focusing on the way in which people are doing the work that they are doing as a feature of the actual work, then we are not building a learning organization.

An effort to create continuous feedback is a better gesture than a yearly review, but this article/video is a fluff piece. If the feedback you're getting in a 360 review is in any way new information, then you're not exhibiting the elements that make a good team in the first place.

If you're attempting to aggrandize feedback behind formality and process, even in the name of immediacy, I think it's counterintuitive.

Feedback needs to be a native mechanism that is implicitly built into the way we are working -- not a sidebar conversation. That's what postmortems are for.

An immediate review system, which in this case sounds like just a forum for immediate complaints/reprimands, sounds terrible because given a negative situation the feedback hasn't had any cool-down time. It sounds very toxic, tactless, lacking introspection, and easily decorated with anger.

We need to be focusing on the context around failures and what the factors were in a situation where a person's actions contributed to a failure. Why did what they were doing make sense to them at the time?

This isn't captured in a 360 review cycle, nor this particular one. I just don't think it fits the tech paradigm of blamelessness/blame-aware at all.

I am struggling to find any reasonable point in this process. Performance reviews in general are more formality than anything in my opinion. At the end of the day I can gauge how my impact is respected across a company on a mgmt/executive level by the money that lands in my bank account. The review cycle from mgmt really does not address any other intricacies.

I've been thinking a lot about this lately, and it might be prudent to take a 'human factors' (Woods, Cook, Dekker, et al) approach to this.

When talking about how it's perhaps more important to investigate the conditions of success over failure, and how the situations contain multiple contributing factors to the way a complex system behaves, one thing that gets highlighted is common ground breakdown between practitioners.

With regard to communication, we are worried about assumptions made, communication fidelity, and the repair process. When we talk about IRC or Slack or Conference calls or email or whatever, those are really just adding another channel of communication and are not necessarily making communication more resilient. Repair processes to communication are a paradigmatic function of good teams.

What makes humans really good at succeeding in high tempo emergency situations is the ability of adaptive capacity.

I think the distinction with regard to remote working can fit into this, albeit with a little bit of corruption and maneuvering of the language.

To speak to this article, the takeaway should be that it's about organizations commiting to a remote-first approach. This goes so far beyond the dicussions we usually think about when wondering wtf people are doing while working from home. It actually also includes the resiliency of working with remote offices globally -- which makes it a culture problem. And the more a change tries to change the culture, the more likely it is that change is going to fail. So remote-first has to be taken by companies on principle or they're just making accomodations and the people who are working remotely are going to suffer. "A bad system will beat a good person every time" for the obligatory Deming quotation. The worst thing a bad process can do is tarnish the repuation of individuals.

Folks whose HQ is in another state/country: do you feel like part of the team or is your influence diminished among the outer rim?

If a team is only simply making accommodations for people to work from home ocasionally or whatever (not truly "remote-first"), then hallway conversations are inexorably changed from a convenience into a liability where people get left out of the loop to develop their own misconceptions about the way things work, the "why's" of what we're doing, and what sort of goals are important.

A remote-first approach can force the organizational infrastructure necessary to make work visible and provide feedback on milestones with distributed, empowered decision making. It is one of the ways (just like TDD, pair programming, etc all attempt to address) to help distribute knowledge across all teams wherever they are to help decrease the heroics needed daily and add ways to address changing demands. Increasing adaptive capacity has to be a fundamental ideal of an organization (aside: the trick here is avoiding building on human misery/heroics, and resulting in burnout).

It also helps improve the onboading process. If it takes 6-12 months for a new teammate to significantly contribute, then that is a risk. An organization with a remote-first approach who can cope with asynchronous workers will have the right kind of documentation, tooling, and relationships between teams to make this robust.

I would highly recommend watching this talk about 'common ground': https://www.youtube.com/watch?v=KgC_N9glqMs

edit:

Let me clarify by that 'remote first' i don't mean force/recommend that everyone work remotely. By remote first, I mean taking the approach that all things should be considering the remote workers in the organization as first class citizens and creating processes that are unable to ignore the aspect of remote working. Pretend that all work could not be done if it wasn't addressing remote working.

In this way, remote working is actually a value add to an organization, not an accommodation.

edit 2:

I've been told my stump speech on this is too much of a millennial idea, which is funny because none of this is my original thoughts. A lot of these ideas come from people who are today aged 50+ or dead.

John Allspaw at Etsy talks about these safety culture luminaries all the time.

If I really wanted to bring up a topic that had passed, I would break out to a separate channel and invite the relevant people and drop a note in the main channel with a "bringing things back up about $conversation in #newchannel". This works better than writing to threads in my experience.

Will be interesting to see how this argument may be affected by life extension technology and if natural death is ever cured.

Irvine, CA has this occurring. 2014 numbers reported that 80% of single family homes in the area were bought by Chinese families (random first google hit article on it: http://www.zerohedge.com/news/2015-09-30/80-all-new-home-buy...). A lot of the Irvine Company apartments top-level units are effectively birthing centers on the down low, but nobody really seems to take much issue with it.

There is also a significant rich foreign Saudi presence in Irvine and that's definitely more of a culture clash due to the misogynistic nature of mother/son relationships. Super spoiled kids, "slavery" scandals, etc but the money "donated" to Irvine company for things like renovating the spectrum mall is massive. Considering Irvine company owns ~90% of Irvine, I think this foreign interest can't be overlooked in lieu of the more public Chinese impact.

All the attempts at 'slack best practices' since this article was written in June 2015 are basically to not do #1, #2, & #3. If Slack is an authoritative source of information that cannot be ignored, you simply aren't doing it right. Slack is a success in my org specifically because it enhances our communication and documentation, not because it has become it.