News for hackers. Stuff that matters.
Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by tauwauwau on Thursday May 05, 2022 @08:20AM (#31272405)
Once we get used to doing same thing multiple times a day, it doesn't matter if the log shows that we're about to take a destructive action, we'll still do it. Only thing that is foolproof is to not take the destructive action because people make mistake, it's human nature. I don't know how this can be implemented, may be encrypt the files, take a backup in some other location (which may not be allowed).

Multiple reviewers here didn't catch the mistake

https://www.bloombergquint.com/markets/citi-s-900-million-mi...

Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by HowardStark on Thursday May 05, 2022 @08:50AM (#31272682)
While this is a huge issue, a solution (well, a partial mitigation) I've seen and used is the "Pointing and Calling" technique. The basic idea is that you incorporate more actions beyond reading and typing or pressing a button—generally by having people point at something and say aloud what it is they're doing and what they expect to happen.

It's used rather extensively in safety-critical public transportation in Japan [1] and to a lesser extent in New York (along with many other countries) [2]. This can easily extend to software without overcomplicating by just setting the expectation that engineers, Q&A, etc. do this even when alone.

[1] https://www.atlasobscura.com/articles/pointing-and-calling-j...

[2] https://en.wikipedia.org/wiki/Pointing_and_calling

Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by samhw on Thursday May 05, 2022 @10:16AM (#31273655)
Hell, GitHub does that to an extent, with the "type the name of this repository to delete it" prompts. Typing the name of the repository isn't exactly perfect, but it's an interesting direction.
Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by Blackcatmaxy on Thursday May 05, 2022 @10:50AM (#31274097)
There was a thread recently about a repo that accidentally went private and lost all of its stars because of confusion with GH teams vs GH profile readme repo naming. I think this type of prompt is very useful for explicitly preventing the rare worst case scenarios but the problem is making any type of prompt "routine" so that our brains fail to process it.
Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by swid on Thursday May 05, 2022 @11:05AM (#31274297)
The suggestion in that post about how to fix it is good, and mirrors one I read in the Rachael by the Bay blog - type the number of machines to continue:

https://rachelbythebay.com/w/2020/10/26/num/

The take away by both is there is actually something to do which can wake people up when the stakes are high, and they might not be doing what they expect.

Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by oauea on Thursday May 05, 2022 @01:22PM (#31276118)
And most importantly, don't let yourself get into the habit of copy pasting the value
Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by underwater on Thursday May 05, 2022 @06:37PM (#31279143)
I wonder if your could print some non visible characters in there to taint the copied value in some detectable way.
Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by dkersten on Saturday May 07, 2022 @12:34PM (#31296097)
Prompt in words, but expect the value in numbers, eg: "Twenty-five" and the box requires you to type "25"? At least this specific case, it would require you to type it.
Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by weaksauce on Thursday May 05, 2022 @08:34PM (#31279837)
yeah, that would possibly stop the copy and paste problem. to make it robust they would need to use a string of a few non-visible characters but that would fail if the browser's clipboard system doesn't copy them over for some kind of privacy initiative. might be another way it fails that I can't think of right now.
Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by lostlogin on Thursday May 05, 2022 @11:02AM (#31274267)
Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by skrtskrt on Thursday May 05, 2022 @06:09PM (#31278918)
I always copy-paste into that box as well, they should probably make at least an attempt at disabling pasting into it
Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by dustymcp on Thursday May 05, 2022 @08:53PM (#31279967)
Azure has the same when deleting a database just a verify this is the correct one by typing the db name
Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by akavel on Thursday May 05, 2022 @02:57PM (#31277178)
I heard of this technique, but unfortunately I don't see how it can be easily applied in software engineering/devops.

Also, I now realized that aviation checklists seem to tend to be done similarly with gestures - at least from what I saw on YouTube, not sure if that's representative or only used during education (?)

Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by samus on Thursday May 05, 2022 @05:27PM (#31278605)
Spelling out loudly the command you are about to execute and explaining the reasoning behind it can help a lot too.
Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by akavel on Friday May 06, 2022 @10:58AM (#31285948)
Ok, but am I to do it on every single command I do on my terminal? Or on which ones specifically? If the problem we're trying to solve is that I can sometimes overlook the "dangerous commands" among "safe ones", by definition of overlooking it won't work if I tell myself to "spell out the command only in case of the dangerous ones", no?

I'm honestly trying to think of the way how I could approach this for myself, just I don't see a clear solution yet that wouldn't require me to spell out everything I type in my terminal window.

Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by emerged on Thursday May 05, 2022 @09:23AM (#31272982)
“I’m removing that semicolon!” (Pointing)
Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by bbarnett on Thursday May 05, 2022 @09:33AM (#31273094)
Parent meant this sort of pointing.

https://t.co/TjfX5K54H7

Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by irrational on Thursday May 05, 2022 @08:49AM (#31272666)
Because everyone assumes that everyone else is looking at it more closely than they are. “I’ll just do a cursory look since I’m sure everyone else is doing a in-depth look.” Narrator: nobody did an in-depth search.
Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by slaymaker1907 on Thursday May 05, 2022 @05:41PM (#31278697)
I'm a fan of doing things temporally so data is very rarely actually deleted from the database. Most of the time, you just update the "valid_to" field to the current time. Sometimes real deleted are required such as with privacy requests, but I think that sort of thing is pretty rare.

If your application has space concerns, you can modify this approach to be like a recycle bin where you delete records which are no longer valid and have been invalid for over a month (or whatever time frame is appropriate for your application). However, I think this is unnecessary in most cases except for blob/file storage.

Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by Danieru on Thursday May 05, 2022 @08:48AM (#31272662)
That form had a couple weird checkboxes with odd wording. It is a famous mistake, but also rather understandable just because the form was cryptic.
Re:I Accidentally Deleted 7TB of Videos Before Going to Prod... by dkersten on Thursday May 05, 2022 @12:05PM (#31275113)
> Multiple reviewers here didn't catch the mistake

Sure, but we can only do so much. I find its good bang for buck and alternatives that might prevent that are not always available, so we do the best we can. You gotta make a call on whether its enough or not.