One morning when I came in and sat down at my desk, all of the old-timers were having coffee and discussing the fiasco. I was very happy to hear all of them talk about how mistakes happen, and the last person to be blamed for such an outage is the poor guy or gal that hit the ENTER button. Rather, blame falls (to various degrees) on: the engineers in their orbit who should be backing them up; the managers helping to onboard them; the chain of command; the entire system that is in place to prevent inappropriate access.
One of my best early-in-career lessons was that it takes maturity to own up to your mistakes (no matter how bone-headed), and it also takes good managers and a good company to foster an environment in which you can own up to them without fear of losing your job. Any company that wants to hang a weight around an intern's neck for something like this is not a company I would want to support in any way.
[0] https://netflixtechblog.com/a-closer-look-at-the-christmas-e...
owning up to your mistakes also gives you an incredible amount of credibility and respect in my opinion. Mistakes happen, especially in complex systems. Owning up to mistakes and explaining your reasoning about your actions makes you and your compatriots better engineers.
Excluding malicious action, most people make a (semi) critical error sometime in their career, especially if you work on the ops side of things, these can often be disasterous. Engineering who claim they never have made a mistake that usually either not working on anything that has value or are just lucky in my opinion. engineering who are afraid to say they made a mistake are a cultural issue aswell, because it delays troubleshooting during incidents.
Something we tell employees during our onboarding in a technical comes down to this.
- reason about a problem by yourself first - think about impact before you do a change, if in doubt, ask and doublecheck. - admit mistakes when you realise them, explain your reasoning and why you did the action. - learn from your mistakes, but accept that being error-free is simply not possible.
If you are in the right company. I made a big mistake at one point. We had all of the people responsible for one of the payment methods away on holiday (this one payment method was managed by another team in another country, as it wasn't using our normal payment gateway and provider that my team was maintaining).
Huge panic, someone needs to fix this, you can't use X any more for completing your order. I'm in the right team that's handling payments and fulfilment, only one at work at that point so I'm told to fix it. I do, I fix it, send it over to testing, get the green light, fix is deployed, everyone is happy.
2 hours later, we figure out the payments are working, but the orders are not being finalised and are still in unpaid. We realised that that single payment method done by this other team in another country was not using our standard payment processing workflow and it has a different way of actually getting the confirmation from the payment provider. This was quite a big company, we had around tens of thousands of Euros blocked in those 2 hours. I own up to it, I admin I made a mistake, go deeper (we did not have anything about this documented) fix it again, we unblock everything, all good. Until 1 month later I got fired (there were layoffs because of larger financial issues, but I was on the list because of the incident), it was the only time in my career this happened to me.
In the same time, someone else in my team made a mistake, hid it from management, even though we knew about it internally, fixed it and bragged about fixing the issue (no mention of him being the one who caused it) and got somehow (didn't even know we had such a thing) employee of the month and big praise in the next department meeting from management.
My lesson from this? Screw these companies and the people running them. I was asked to help, I did it, I made a mistake, fully aware of that, but then I'm the only one thrown under the bus for it.
To give you a counterexample.
The same company my prior example came from, also had some other "silly mistakes" made by an intern. He had to do inventory of a couple of old servers and remove hard disks from these servers. The servers where to be sold.
Sadly, no one told him we had additional servers in the back of the storage room which he forgot to check because they where not on the same pallet as the batch he was told to check.
Result, an couple of servers got sold with disks still in them. Luckely the company we sold to was friendly enough to give us a headsup about it and it resulted in no further issues, but still. Our company director personally took this as a reason to spearhead a plan about improving operational security and change processes (Aka, remove the hard disks when the machines are put out of service instead of half a decade later when their sold).
The intern felt pretty bummed and thought he was responsible for the mistake, but in my opinion he done the job that was asked of him, he just got incomplete instructions. This was also explicitly communicated with him by his direct supervisor.
In my experience, not throwing people under the bus to hide organizational or process failure, but simply admitting the processes could be better and striving for improvement does absolute wonders for morale and team building.
Being perfect is impossible, organizations should keep people to impossible standards, especially to hide incompetence.
Third option: they don't realise when they make a mistake, either because they are not smart enough or too full of themselves.
The only way to guaranteed never have mistakes in your code, is to not have any code at all.
Bingo. I’ve made mistakes that have taken down systems or caused them to silently fail. The worst mistake I’ve made took down basically my entire company for about 20 minutes. This turned out not to be critical, because our site was still operating, and it was just external data feeds that weren’t getting updated, but I freaked out about it for a minute. After that minute, I went and got help, and we fixed it. Had I not, I probably could have fixed it myself, but it would have taken much longer and cost much more than it did.
If you’re in an environment that doesn’t recognize that, you aren’t in a place that actually values and understands engineering work.
That outage is what drove us to rearchitect all of Netflix to be multi-region.
Any customer that can see through the BS will immediately turn around on their heels. And SolarWinds will be happy that they just lost that problem customer.
SolarWinds is looking for an Equifax not a Netflix.
Every individual contributor in that company has just learned that they need to cover their ass for any action that could possibly go wrong.
The cultural outcome is that accountabilities will be spread across managers so that blame can’t be assigned to an individual.
If there were any singular behavior I wish every tech company would adopt, it is that level of perspective on what makes a system reliable.
At the time, I had several years in the industry under my belt but was new to big tech companies, and especially the tech that Amazon had built. I lacked confidence in myself but I was super conscious of actively seeking out the help of others. And it failed spectacularly - one SDE3 in my team with whom I was sort of paired would quite literally back away slowly when I was showing him an issue I needed his advice on.
Big companies like that have such variation in teams, and I think I just got rather unlucky in my experience.
I have an employee who I thought I could give more responsibilities, but he keeps not locking his computer when he walks away. He has very limited access to everything and it would impede his career if he didn’t also have the same attitude about other issues. (My question is, how do I make him diligent — It’s real potential wasted).
We had a similar thing at one gamedev place that I worked at where and email would go out to the team if you left your computer unlocked(I forget the exact phrase but it was fairly silly).
We had shared offices and one of the programmers had the office right next to the kitchen. One day we all heard the senior programmer shout "WHAT THE FUCK!" and all ran over to see what had happened.
It turns out one of our engineers had walked into the kitchen and left his computer unlocked. The senior developer seeing this had opened up outlook, started a new message and began typing in the subject. What he didn't know is the developer had hand-rolled a keylogger with a match pattern for the message that everyone would send and dispatched Windows+L via key injection to the main window loop.
The trap was sprung and the machine locked right in front of him as he typed the last letter unable to send the email.
There was all sorts of other shenanigans at that place(like a fake "April 2nd" firing, they got the person who did that back with an annoy-a-tron over a 6 month period) but that was one of the more memorable ones.
Edit: Also singing over the top praises of employees using the victim's account. "MonkeyButton is truly the best coworker I have ever had "
All this fun has gone away now with Covid and remote working.
I think that there's sometimes a tendency to overthink management topics - there's no need to overthink this one.
Physical access is considered game over, no?
For example, posted yesterday: https://tailscale.com/blog/rotate-ssh-keys/ (https://news.ycombinator.com/item?id=26249380)
> One company had an employee they fired for bad behaviour. A few months later all their production servers got wiped out. ... A month later, it happened again! Everything gone.
> What happened was that the rogue employee, months before they were fired, had waited until everyone went to lunch, walked around from one PC to the next, and collected the ssh private keys from any of them that weren’t locked. ... Then, a while after getting fired, he used a co-worker’s ssh key to login and destroy everything.
Of course, screen locking would not prevent a determined, disgruntled (or desperate) employee from pulling off something like this, but it would stop the casual collection of secrets while everyone is at lunch. Add full disk encryption to PCs and you have a reasonable chance at defending.
Oh, and if you're doing anything with credit cards, PCI DSS requires you to enforce a workstation locking policy. Failing to do so opens you up to massive liability in the event of a breach.
I wish I had an answer for the second part; it's hard to see someone with talent be the one to get laid off after months of asking them to be more diligent in their work.
Oof, this hits close to home. I don't have an answer either.
If nothing else at least it promotes awareness (and cake!).
It's reasonably believable that they leaked their password because. Interns.
And the second factor?
It might be that in your organisation this is a cargo-cult security practice that he's not bothered by because he knows it's not an effective practice.
Or it could be that he knows he doesn't have enough permissions to do any damage, so he doesn't bother locking his computer. Trusting him with a little responsibility might change that.
If a mistake happens, don't blame the individual, blame the process then find a way to fix that process. If a company has a blame culture people spend more time covering their own arse instead of building safer processes.
"you can delegate authority, but you cannot delegate responsibility." [1]
This CEO is a very poor leader.
1. https://www.theleadermaker.com/you-cant-delegate-responsibil...
1) access restrictions such that even malicious interns, and certainly careless interns can do little damage *when* the inevitable leak happens
2) Actively scan everyone's online presence, and let them know that this is a requirement of employment.
3) Require 2FA
4) Much better training so it is reduced
5) Internally firewalled and airgapped systems
I could go on... but none of these were done
The fact that they blame the intern shows that they are insanely unqualified for any job related to any sort of security. These CxOs are active hazards in the industry.
this should be something that is implemented in any organisation beyond a very small scale, mainly because even if not malicious, people should not be able to make critical mistakes in systems they have no know how off.
The intern leaked the password, but how was he able to know this was critical information? Not to mention he should never have been put in that position in the first place.
I've worker with interns whose major redeeming quality was that their internship was fixed length and would be over soon. I've also worker with interns who demonstrated ability and responsibility sufficient to get the same access that I had (and, clearly, an offer letter). That it was an intern doesn't mean the level of access was inappropriate; of course, if it were my intern, I would take blame for them leaving their password on github.
1) Every password policy allows for dumb passwords in certain places. Because password policies are only enforced on systems that integrate with the password policy enforcement mechanism. Which never covers everything. Even with a password policy, it’s easy to make dumb passwords.
2) It doesn’t say that the intern chose this policy or that it belonged to something critical. There has been no link established between that password and the breach. A random researcher said they found the password a year ago and reported it. It could have been used, but there’s no reason to believe it’s relevant.
3) Nothing suggests that they did.
Conclusion: It’s better to not be an armchair quarterback after a breach, especially when it’s still under active investigation by actual professionals with access to actual data, and they aren’t even making the claims that folks here are making.
Imagine the size of a rainbow table of SHA512 or SHA3-512 hashes for a 60 character password. A 60 character password could be as simple as:
* I enjoy driving my tiny white car with a standard transmission.
* My big cat, Ace, really sleeps a lot during the day while I work!
* Growing up my favorite song was Time by Pink Floyd about regret :(
For the few that you need to manually enter, something long is good, but it should ideally use characters or different classes, and ideally not be comprised solely of dictionary words (which your first example is), otherwise the search space is greatly reduced.
Also, manually entering a 60 character password is not going to be fun :) I think the longest "manual enter" password I have is 25 characters, and it's a PITA to enter in a password field!
You only need that to impose a greater character width against brute force attacks. That is the only value in high entropy.
The actual reason people think they need this is because it was written into a NIST publication a very long time ago and it just became common practice. As a proof of concept what is the published standard that imposes that practice? I bet you think you need this but cannot find the written standard guidance suggesting it.
The guy who originally wrote that standard later came out and said it was a mistake. Bad advise that he wishes he could take back, but it’s too late everybody thinks they need it and they don’t know why or where that guidance even comes from.
> and it's a PITA to enter in a password field!
Only on a touch screen.
My first example also contains uppercase and punctuation. Think of it like IPv6. When the key space is large enough you don’t need a bunch of bullshit and gimmicks to ensure uniqueness.
That's exactly the reason I meant - to make brute forcing take much longer.
> Only on a touch screen
Yep, certainly worse on a touch screen, but I hate entering long passwords even on a keyboard - it's OK if it's a field that let's you see what you type, but if it's just asterisks, then I find it hard not seeing that visual feedback.
So let's analyze the various scenarios under which the "intern" might have been responsible, and why each one is bullshit.
1) The intern exposed a company password on their own account.
Counter: What kind of "password policies" allowed such a weak password in the first place?
2) The intern came up with the weak password themselves, thus violating "password policies" not just for secrecy but strength/security. This password was then used for several critical, production applications.
Counter: Why was an intern in charge of deciding a password used for anything critical?
3) The intern came up with the weak password and exposed it, but it was only used for the intern's own corporate accounts (e.g. their Windows workstation).
Counter: Why did an intern have a level of access such that their account being breached could lead to this level of compromise/exfiltration?
Conclusion: There is no conceivable scenario under which this makes sense.