News for hackers. Stuff that matters.
Re:Ask HN: Ever worked with a service that can never be rest... by _jal on Thursday January 16, 2020 @09:51AM (#22064991)
Back when I ran a company, my partners and I gamed out what to do with employees in various situations, as a way of coming to a consensus. I was against legal action in all but extreme hypotheticals. Behavior like this is in that class.

You assume you not only have all the relevant information to predict the outcome, but also that your analysis of the situation is superior enough to trump those who actually own the process and will be responsible for the aftermath of your destructive actions. You demonstrate you know this by suggesting stealth.

Thinking about the fact that people like this are out there makes me paranoid about my hiring all over again.

Re:Ask HN: Ever worked with a service that can never be rest... by celticmusic on Thursday January 16, 2020 @10:12AM (#22065201)
If you had hired competent people to begin with, the configs would have been backed up.

I don't agree with the other posters recommendation, but I understand the sentiment. The same incompetence that allowed the event to happen is the same incompetence that allowed it to be swept under the rug for almost 6 years.

I've seen paragons of hiring get upset because failures were made visible, not understanding that the failures were there all along and they were playing a game of chicken with reality, only now they have a chance of winning.

Re:Ask HN: Ever worked with a service that can never be rest... by setr on Thursday January 16, 2020 @10:33AM (#22065440)
>If you had hired competent people to begin with, the configs would have been backed up

What happy-go-lucky universe do you hail from? You could hire the most competent people in the world and you'll still end up with a horribly complex shitshow with lost programs, untested backups, missing docs, implicit information out the ass, and so on. Software development has never had the discipline to claim compotent employees produce competent results -- they're just less likely to be shitty results.

And we spend 90÷ of our time dealing with every layer of incompetence our competent, and incompetent, friends have provided us

Re:Ask HN: Ever worked with a service that can never be rest... by celticmusic on Thursday January 16, 2020 @10:48AM (#22065569)
It's possible to avoid losing a configuration.
Re:Ask HN: Ever worked with a service that can never be rest... by setr on Thursday January 16, 2020 @11:19AM (#22065947)
It's possible, and it's possible to avoid losing source code, and its.possible to have valid backups, and keep your servers decemtly secure... yet it's sufficiently common (especially as the program ages) that its not trivially attributed to competency; rather you need continuous, unfailing competency, in the face of continuous environmental change (local changes, within the business, and external changes, as business needs change, and mergers occur, and people quit/fired, etc)
Re:Ask HN: Ever worked with a service that can never be rest... by workingpatrick on Friday January 17, 2020 @01:54PM (#22077862)
It's more than possible, it's expected. While all of these things do happen, they are preventable and should not be treated as excusable, people need to be held accountable and systems / processes need to be put in place to ensure that these things don't reoccur.
Re:Ask HN: Ever worked with a service that can never be rest... by celticmusic on Thursday January 16, 2020 @11:33AM (#22066089)
> rather you need continuous, unfailing competency

That's called doing it manually.

What you're doing is making excuses rather than holding yourself to a higher level of quality.

Re:Ask HN: Ever worked with a service that can never be rest... by setr on Thursday January 16, 2020 @01:31PM (#22067385)
>That's called doing it manually.

Yes, a common feature in unstable environments. Automation is an ideal, but not a necessity, and it's not always viable or cost-effective.

But also that the unfailing competency means that you need to contiously maintain (all aspects of the) automation without fail, again in spite of a changing environment. Across everyone who does things.

>What you're doing is making excuses rather than holding yourself to a higher level of quality.

And what you're doing is pretending that firing off blind, trivial prescriptions is a useful activity -- perhaps I'll soon find you selling self-help books as well :-)

Automation is not an answer that can always be applied trivially; especially retroactively. Even despite all the marketing that will tell you otherwise.

Re:Ask HN: Ever worked with a service that can never be rest... by celticmusic on Thursday January 16, 2020 @02:32PM (#22067989)
> Automation is an ideal, but not a necessity

You're just bad. You automate specifically to take the human error risk out of the equation, that's a part of what it means to be competent. We don't automate our deploys because it's an ideal that is not a necessity, but so some jackass doesn't delete a configuration while also never putting it into a backup, or source control, or on their dev machine, ad nauseum.

It's not an accident that some of us have been working 20+ years in this industry and have never been the cause of something like this.

> Automation is not an answer that can always be applied trivially; especially retroactively. Even despite all the marketing that will tell you otherwise.

shit, I guess I didn't read the right pamphlet...

Re:Ask HN: Ever worked with a service that can never be rest... by nnq on Thursday January 16, 2020 @05:33PM (#22069750)
> Thinking about the fact that people like this are out there makes me paranoid about my hiring all over again.

To be honest, I was 100% trolling or troll-baiting "for the lulz" - I wouldn't actually ever apply my "advice". And I don't think there's actually a significant chance of coming across people thinking this way in your business unless you work in very specific areas.

I mean, sure, some of us enjoy to destroy/break things or to make them fail or fall apart for either "the fun of it" or "to leave space for future better things", but we do it in "special controlled environments" to get our kicks, not in our business/work. Unless you're in the military or a similar business you wouldn't ever be in the position or hiring people expected to manifest their selective-destructive tendencies at work and have the risk of them also occasionally turning the gun in the wrong direction.

The biggest problem for a business imo is people who actually do this but do it subtly and unconsciously: like the developers picking "interesting" architectures and technologies that end up destroying the possibility of integration with legacy services and indirectly forces an expensive rewrite of those legacy systems, the managers that adopt "novel" management systems that end up pushing valuable people out of the company etc.

People that are consciously doing sabbotage and destructive action (even if it's "to make the world a better place" in their views) are easy to spot and handle (eg. legally), and those with the competence to really do things stealthly are always more motivated to apply such skills in other areas... the other "damaged apples" doing it without fully realizing are dangerous for businesses and you need to worry about them when hiring...