News for hackers. Stuff that matters.
Re:Ask HN: Ever worked with a service that can never be rest... by nnq on Thursday January 16, 2020 @09:16AM (#22064677)
Unpopular advice: find a subtle way to make it crash, preferably stealthy, but if not possible, at least in a way that can be attributable to mild innocent incompetence instead of malice!

Then there will be more and more interesting work to do for you and others, either rediscovering and properly documenting the config, or, hopefully, architecting and coding its replacement! In the aftermath, the organization will be more robust. If it actually collapses bc of this, then it deserved to die anyway, you only helped accelerate the outcome and reduced the suffering.

Some things and processes need to be "helped to fail faster", everyone will benefit from the renewal in the end, even if most will hate it ;)

Re:Ask HN: Ever worked with a service that can never be rest... by _jal on Thursday January 16, 2020 @09:51AM (#22064991)
Back when I ran a company, my partners and I gamed out what to do with employees in various situations, as a way of coming to a consensus. I was against legal action in all but extreme hypotheticals. Behavior like this is in that class.

You assume you not only have all the relevant information to predict the outcome, but also that your analysis of the situation is superior enough to trump those who actually own the process and will be responsible for the aftermath of your destructive actions. You demonstrate you know this by suggesting stealth.

Thinking about the fact that people like this are out there makes me paranoid about my hiring all over again.

Re:Ask HN: Ever worked with a service that can never be rest... by celticmusic on Thursday January 16, 2020 @10:12AM (#22065201)
If you had hired competent people to begin with, the configs would have been backed up.

I don't agree with the other posters recommendation, but I understand the sentiment. The same incompetence that allowed the event to happen is the same incompetence that allowed it to be swept under the rug for almost 6 years.

I've seen paragons of hiring get upset because failures were made visible, not understanding that the failures were there all along and they were playing a game of chicken with reality, only now they have a chance of winning.

Re:Ask HN: Ever worked with a service that can never be rest... by setr on Thursday January 16, 2020 @10:33AM (#22065440)
>If you had hired competent people to begin with, the configs would have been backed up

What happy-go-lucky universe do you hail from? You could hire the most competent people in the world and you'll still end up with a horribly complex shitshow with lost programs, untested backups, missing docs, implicit information out the ass, and so on. Software development has never had the discipline to claim compotent employees produce competent results -- they're just less likely to be shitty results.

And we spend 90รท of our time dealing with every layer of incompetence our competent, and incompetent, friends have provided us

Re:Ask HN: Ever worked with a service that can never be rest... by celticmusic on Thursday January 16, 2020 @10:48AM (#22065569)
It's possible to avoid losing a configuration.
Re:Ask HN: Ever worked with a service that can never be rest... by setr on Thursday January 16, 2020 @11:19AM (#22065947)
It's possible, and it's possible to avoid losing source code, and its.possible to have valid backups, and keep your servers decemtly secure... yet it's sufficiently common (especially as the program ages) that its not trivially attributed to competency; rather you need continuous, unfailing competency, in the face of continuous environmental change (local changes, within the business, and external changes, as business needs change, and mergers occur, and people quit/fired, etc)
Re:Ask HN: Ever worked with a service that can never be rest... by workingpatrick on Friday January 17, 2020 @01:54PM (#22077862)
It's more than possible, it's expected. While all of these things do happen, they are preventable and should not be treated as excusable, people need to be held accountable and systems / processes need to be put in place to ensure that these things don't reoccur.
Re:Ask HN: Ever worked with a service that can never be rest... by celticmusic on Thursday January 16, 2020 @11:33AM (#22066089)
> rather you need continuous, unfailing competency

That's called doing it manually.

What you're doing is making excuses rather than holding yourself to a higher level of quality.

Re:Ask HN: Ever worked with a service that can never be rest... by setr on Thursday January 16, 2020 @01:31PM (#22067385)
>That's called doing it manually.

Yes, a common feature in unstable environments. Automation is an ideal, but not a necessity, and it's not always viable or cost-effective.

But also that the unfailing competency means that you need to contiously maintain (all aspects of the) automation without fail, again in spite of a changing environment. Across everyone who does things.

>What you're doing is making excuses rather than holding yourself to a higher level of quality.

And what you're doing is pretending that firing off blind, trivial prescriptions is a useful activity -- perhaps I'll soon find you selling self-help books as well :-)

Automation is not an answer that can always be applied trivially; especially retroactively. Even despite all the marketing that will tell you otherwise.

Re:Ask HN: Ever worked with a service that can never be rest... by celticmusic on Thursday January 16, 2020 @02:32PM (#22067989)
> Automation is an ideal, but not a necessity

You're just bad. You automate specifically to take the human error risk out of the equation, that's a part of what it means to be competent. We don't automate our deploys because it's an ideal that is not a necessity, but so some jackass doesn't delete a configuration while also never putting it into a backup, or source control, or on their dev machine, ad nauseum.

It's not an accident that some of us have been working 20+ years in this industry and have never been the cause of something like this.

> Automation is not an answer that can always be applied trivially; especially retroactively. Even despite all the marketing that will tell you otherwise.

shit, I guess I didn't read the right pamphlet...

Re:Ask HN: Ever worked with a service that can never be rest... by nnq on Thursday January 16, 2020 @05:33PM (#22069750)
> Thinking about the fact that people like this are out there makes me paranoid about my hiring all over again.

To be honest, I was 100% trolling or troll-baiting "for the lulz" - I wouldn't actually ever apply my "advice". And I don't think there's actually a significant chance of coming across people thinking this way in your business unless you work in very specific areas.

I mean, sure, some of us enjoy to destroy/break things or to make them fail or fall apart for either "the fun of it" or "to leave space for future better things", but we do it in "special controlled environments" to get our kicks, not in our business/work. Unless you're in the military or a similar business you wouldn't ever be in the position or hiring people expected to manifest their selective-destructive tendencies at work and have the risk of them also occasionally turning the gun in the wrong direction.

The biggest problem for a business imo is people who actually do this but do it subtly and unconsciously: like the developers picking "interesting" architectures and technologies that end up destroying the possibility of integration with legacy services and indirectly forces an expensive rewrite of those legacy systems, the managers that adopt "novel" management systems that end up pushing valuable people out of the company etc.

People that are consciously doing sabbotage and destructive action (even if it's "to make the world a better place" in their views) are easy to spot and handle (eg. legally), and those with the competence to really do things stealthly are always more motivated to apply such skills in other areas... the other "damaged apples" doing it without fully realizing are dangerous for businesses and you need to worry about them when hiring...

Re:Ask HN: Ever worked with a service that can never be rest... by setr on Thursday January 16, 2020 @10:22AM (#22065305)
This is naive evil Mastermind level thinking, where the villain spouts his whole line and you think "no wait that's just stupid.." as he sends off the sequence of nukes

>If it actually collapses bc of this, then it deserved to die anyway, you only helped accelerate the outcome and reduced the suffering

If you want to make an evolutionary "survival of the fittest" type of claim... Life is entirely about error correction (and reproduction + staving off death as long as possible). Engineers babbling to engineers is part of a business's error correction.. Duct tape and monkeypatching is part of the engineer's error correction..

A rogue engineer trying to invoke rapid deconstruction of his environment (a virus) is one of those errors -- Solution: engage, restrain, and eject.

The legitimate error correction is already happening, by means of hope & replace. Presumably because other business concerns make it so that actual shutdown would be a Terrible, Horrible, No Good, Very Bad idea.

>If it actually collapses bc of this, then it deserved to die anyway, you only helped accelerate the outcome and reduced the suffering.

A doctor can't help you if you die in minutes. They can save you if you die in hours. Time is a major part of error correction.

Oh damn, you've pushed out the nukes already.

>Some things and processes need to be "helped to fail faster", everyone will benefit from the renewal in the end, even if most will hate it ;)

Seems to me the main process requiring failure acceleration is this one employee's UAC expiration policy.

Re:Ask HN: Ever worked with a service that can never be rest... by workingpatrick on Friday January 17, 2020 @01:52PM (#22077825)
Came here to post something similar. Why jump through all these hoops to unfuck some inherently fucked system.

It's a bunch of work and time and effort to reverse engineer it just to get back to the point of still having a piece of shit legacy system.

Honestly this is effectively natural selection at play for businesses. They put themselves in this situation, if it dies and destroys the business ... good fucking riddance, the people who let this situation occur need to fail hard, it's the only way they will ever learn.

Just walk up to the machine and plug in a USB kill for 2 seconds, remove it, and get on with your life.