This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don't actually do it. Then have multiple people inspect the logs. Once ok'd, run again, with manual prompts after each log item asking to continue, for the first few files/bits of data. Only after that was ok'd too did it run the remainder.
In other things I've worked on, I've taken the terraform-style plan first, then apply the plan approach, with manual inspection of the plan in between.
Multiple reviewers here didn't catch the mistake
https://www.bloombergquint.com/markets/citi-s-900-million-mi...
It's used rather extensively in safety-critical public transportation in Japan [1] and to a lesser extent in New York (along with many other countries) [2]. This can easily extend to software without overcomplicating by just setting the expectation that engineers, Q&A, etc. do this even when alone.
[1] https://www.atlasobscura.com/articles/pointing-and-calling-j...
https://rachelbythebay.com/w/2020/10/26/num/
The take away by both is there is actually something to do which can wake people up when the stakes are high, and they might not be doing what they expect.
Also, I now realized that aviation checklists seem to tend to be done similarly with gestures - at least from what I saw on YouTube, not sure if that's representative or only used during education (?)
I'm honestly trying to think of the way how I could approach this for myself, just I don't see a clear solution yet that wouldn't require me to spell out everything I type in my terminal window.
If your application has space concerns, you can modify this approach to be like a recycle bin where you delete records which are no longer valid and have been invalid for over a month (or whatever time frame is appropriate for your application). However, I think this is unnecessary in most cases except for blob/file storage.
Sure, but we can only do so much. I find its good bang for buck and alternatives that might prevent that are not always available, so we do the best we can. You gotta make a call on whether its enough or not.
For database entries, flag for deletion, then delete.
In the files case, the move or rename also accomplishes the result of breaking any functionality which still relies on those file ... whilst you can still recover.
Way back in the day I was doing filesystem surgery on a Linux system, shuffling partitions around. I meant to issue the 'rf -rm .' in a specific directory, I happened to be in root.
However ...
- I'd booted a live-Linux version. (This was back when those still ran from floppy).
- I'd mounted all partitions other than the one I was performing surgery on '-ro' (read-only).
So what I bought was a reboot, and an opportunity to see what a Linux system with an active shell, but no executables, looks like.
Plan ahead. Make big changes in stages. Measure twice (or 3, or 10, or 20 times), cut once. Sit on your hands for a minute before running as root. Paste into an editor session (C-x C-e Readline command, as noted elsewhere in this thread).
Have backups.
And yes, copy, verify, delete. And make sure by the code structure that you either do the three on the same files, or their fail.
Also, do it slowly, with just a bit of data on each iteration. That will make the verification step more reliable.
Anyway, for a huge majority of cases, only having backups is enough already. Just make sure to test them.
Example:
cd datadir
mkdir delete
mv <list of files to be deleted> ./delete
# test to see if anything looks broken.
# This might take a few seconds, or months, though it's usually reasonably brief.
rm -rf ./delete
- It's atomic (on a single filesystem). There's no risk of ending up with a partial operation or an incomplete operation.
- It doesn't copy the data, it renames the file. (mv and rename are largely synonyms.)
- There's no duplication of space usage. Where you're dealing with large files, this is helpful.
The process is similar to the staged deletion most desktop OS users are familiar with, of "drag to trash, then empty trash". Used in the manner I'm deploying it, it's a bit more like a staged warehouse purge or ordering a dumpster bin --- more structured / controlled staged deletion than a household or small office might use.
> ... Then have multiple people inspect the logs. Once ok'd, run again, with manual prompts after each log item asking to continue...
The process was for TWO of us to sit in front of computer and review the curves in this crazy old EDA layout tool called "L-edit" before it got sent to have the actual masks made (which were very expensive). It took HOURS to check everything.
The first hour was tolerable but then boredom started to creep in and we got sloppy. The whole reason TWO people got tasked with this was because it was thought that we would keep each other focused-- 2 pairs of eyes are better than one, right?. Instead, it just underscored the tedium of it all. One day someone walked in and found us BOTH in DEEP SLEEP in front of the monitor. Having two people didn't decrease the waste caused by mistakes, it just bored the hell out of more people.
Was it worth it? No, I don't think so from an opportunity cost perspective-- even though we were the most junior folks there. A mind is a terrible thing to waste!
I think that this is the most important part of any check. Your parent refers to checking the log five times, but, at least in my experience, I won't catch any more errors on the fifth time than the first—if I once saw what I expected rather than what was there, I'll keep doing so. Of course everyone has their blind spots, but, as in the famous Swiss-cheese approach, we just hope that they don't line up!
See PDCA for more a more time critical decision loop. https://en.wikipedia.org/wiki/PDCA
I was running commands manually to interact with files and databases, but was quickly shown that even just writing all the commands out, one by one gives room personally review and get a peer review, and also helps with typos. I could ask a colleague "I'm about to run all these commands on the DB, do you see any problem with this?". It also reduces the blame if things go wrong if it managed to pass approval by two engineers.
While I'm thinking back, another little tip I was told was to always put a "#" in front of any command I paste into a terminal. This stops accidentally copying a carriage return and executing the command.
For a one-liner sure, but a multi line command can still be catastrophic.
Showing the contents of the clipboard in the terminal itself (eg via xclip) or opening an editor and saving the contents to a file are usually better approaches. The latter let’s you craft the entire command in the editor and then run it as a script.
[For Bash] Ctrl + x + Ctrl + e : launch editor defined by $EDITOR to input your command. Useful for multi-line commands.
I have tested this on windows with a MINGW64 bash, it works similarly to how `git commit` works; by creating a new temporary file and detecting* when you close the editor.
[0] https://github.com/onceupon/Bash-Oneliner
* Actually I have no idea how this works; does bash wait for the child process to stop? does it do some posix filesystem magic to detect when the file is "free"? I can't really see other ways
Once that happens, it reads from the temporary file that it created.
It will prevent any number of newlines from running the commands if they're pasted instead of typed.
You can enable it either in .inputrc or .bashrc (with `bind 'set enable-bracketed-paste on'`)
Anyway, I selected what I though was a "merge all duplicates" option without previewing results. What I had actually done was "merge all selected". So, the system proceeded to merge a very large % of the database... Into One. Single. Record.
Luckily the vendor kept very good backups, and so I kept my job. Because I also luckily had a very good boss and I had already demonstrated my value in other ways, he just asked me "Well, are you going to make that mistake again?". I wisely said no, and he just smiled and said "Then I think we're done here."
I have been particularly fortunate throughout my career to have very good managers. As much as managers get a lot of flack here on HN, done well they are empowering, not a hindrance, and I attribute a lot of success in my career to them.
I think that, if you've only learned something like that the easy way, then you haven't learned it yet. As long as everything's only ever gone right, it's easy to think, I'm in a rush this one time, and I've never really needed those safety procedures before, ….
It's simple to manually test corner cases, and then when everything is smooth I can just
script1 | xargs script2
When I'm trying to get script1 right I can pipe it to a file, and cat the file to work out what the next sed or awk script needs to be.
Over-simplified example:
1. Copy stuff from A to B
2. Delete stuff from A
(Obviously you wouldn't do it like that, but just for illustration purposes.) It's all fine, but (2) assumes that (1) succeeded. If it didn't, maybe no space left, maybe missing permissions on B, whatnot, then (2) should not be executed. In this simple example you could tie them with `&&` or so (or just use an atomic move), but let's say these are many many commands and things are more complex.
You could always generate an intermediary set, inspect/test/etc, and then apply it with Python. I've done that too, works just as well. The important thing is to separate the planning step from the apply step.
* where "complicated" means more complicated than, for ex, `rm some_path.txt` or `DELETE FROM table WHERE id = 123`.
- make sure the sum of their lengths == number of total current items
- make sure items_to_be_kept.length != 0
- make sure no two items appear in both lists
- check some items chosen at random to see if they were sorted in the correct list
At this point the only possible mistake left is to confuse the lists and send the "to_be_kept" one to the delete script; a dry run of the delete list can be in order.
This is the most reliable way. Bash has a few niceties for error handling, but if you are using them, you would probably fare better in another language.
If you do insist on Bash, quote everything, and use the "${var}" syntax instead of "$var". Also, make sure you handle every single possible error.
If you _absolutely_ must use a shell script:
0. Use shellcheck, which will warn you about many of the below issues: https://www.shellcheck.net/
1. understand how quoting and word splitting work: https://mywiki.wooledge.org/Quotes
2. if piping files to other programs, using `-print0` or equivalent (or even better, if using something like find, its built in execution options): https://mywiki.wooledge.org/UsingFind
3. Beware the pitfalls (especially something like parsing `ls`): https://mywiki.wooledge.org/BashPitfalls
(warning: the community around that wiki can be pretty toxic, just keep that in mind when foraying into it.)
It's like the difference between "do what I say" and "do what I mean"...
- -WhatIf to see what would happen if you run the script;
- -Confirm, which asks for confirmation before any potentially destructive action.
Moreover these arguments get passed down to any command you write in your script that support them. So you can write something like:
[CmdletBinding(SupportsShouldProcess)]
param ([Parameter()] [string] $FolderToBeDeleted)
# I'm using bash-like aliases but these are really powershell cmdlets!
echo "Deleting files in $FolderToBeDeleted"
$files = @(ls $FolderToBeDeleted -rec -file)
echo "Found $($files.Length) files"
rm $files
I can also declare that my script is "High" impact with the "ConfirmImpact = High" switch. This will make it so that the user gets asked for confirmation without explicitly passing -Confirm. A user can set their $ConfirmPreference to High, Medium, Low, or None, to make sure they get asked for confirmation for any script that declare an impact at least as high as their preference.
[1]: https://docs.microsoft.com/en-us/powershell/scripting/learn/...
Cause if it is an entirely separate code path, doesn’t that introduce a case where what you say you’ll isn’t exactly what actually happens?
> because I didnt read the docs
Ouch.
> Or is it a separate routine that you have to write?
If you are writing a function or a module what would do something (eg API wrapper) then of course you need to write it yourself.
But if you are writing just a script for your mundade one-time/everyday tasks and call cmdlets what supports ShouldProcess then it works automagically. Issuing '-whatif' for the script would pass `-whatif` to any cmdlet what has 'ShouldProcess' in it's definition. Of course if someone made a cmdlet with a declared ShouldProcess but didn't write the logic to process it - you are out of luck.
But if have a spare couple of minutes check the docs in the link, it was originally a blog post by kevmarq, not a boring autodoc.
Your example is changing the list of files, or arguments to rm between runs. Same as pc85’s example where the timestamp argument changes.
If you run rm * multiple times, the state of the system changes each time because that ‘timestamp’ ends up being different each time.
But if instead you run an rm on files older than a fixed timestamp, multiple times, the resulting filesystem is idempotent with respect to that operation, because the timestamp ends up set to the same value, and the filesystem in every case contains all the files added later than that timestamp.
if * expands to the rm binary itself, maybe.
# rm rm
rm: command not found
T'was a sad day.
This pattern has the added benefit that it makes it really easy to write unit tests, which is something often sorely lacking in these sorts of batch scripts. It also makes full automation down the line a breeze, since you have nice shearing layers between your components.
https://news.ycombinator.com/item?id=29083367
It defaults to not doing anything so you can gradually and selectively have it do something.
Learned about when I posted my command line checklist tool on HN: https://github.com/givemefoxes/sneklist
(https://news.ycombinator.com/item?id=25811276)
You could use it to summon up a checklist of to-dos like "make sure the collection in the dictionary has the expected number of values" before a "do you want to proceed? Y/n"
(step1) Download video from URL. Include the Id in the filename.
(step2) Grab the list of files that have been downloaded and parse to get the Id. Using the Id, delete the original file.As they say, measure twice, cut once.
Don't feel bad, I think every professional in IT goes through something similar at one time or another.
Decide, then act.
Another of my favorites is egregious use of caching, because traversing a DAG can result in the same decision being made four or five times, and the 'obvious' solution is to just add caches and/or promises to fix the problem.
As near as I can tell, this dates back to a time when accumulating two copies of data into memory was considered a faux pas, and so we try to stream the data and work with it at the same time. We don't live there anymore, and because we don't live there anymore we are expected to handle bigger problems, like DAGs instead of lists or trees. These incremental solutions only work with streams and sometimes trees. They don't work with graphs.
Critically, if the reason you're creating duplicate work is because you're subconsciously trying to conserve memory by acting while traversing, then adding caches completely sabotages that goal (and a number of others). If you build the plan first, then executing it is effectively dynamic programming. Or as you've pointed out, you can just not execute it at all.
Plus the testing burden is so drastically reduced that I get super-frustrated having to have this conversation with people over and over again.
Automated tests are awesome :)
During buildup of the our_id list: assert (vimeoId not in our_ids).
After creating the list: assert len(set(our_ids)) > 10000 and assert len(set(our_ids)) == len(our_ids)
Before each final deletion: assert id not in hardcoded_list_of_golden_samples.
Depending on the speed required you could hit the api again here as an extra check.
Yes, that can be a simple but powerful live on screen log. I developed a library to use an API from a SaaS vendor, in much the same way as the author. It was my first such project & I learned the hard way (wasted time, luckily no data loss or corruption) that print() was an excellent way to keep tabs on progress. On more than one occasion it saved me when the results started scrolling by and I did an oh sh*t! as I rushed to kill the job.
Before: there is chance there is a bug in my "delete" use case
Now: what we have before plus the change that there is a bug in my "--live-run" flag
If both paths are on the same disk moving files is a fast operation - and if you discover a screw up, you can easily undo it. On the other hand if everything still looks fine after a few days, you just `rm -rf` that folder and purge the files.
Instead of performing the dangerous action outright, just log a message to screen (or elsewhere) and watch what is happening.
Alternatively, or subsequently, chroot and try that stuff on some dummy data to see if it actually works.
Make sure you got everything out and off before you pull up your pants, or else you better be prepared to deal with all the shit that might follow!
SELECT COUNT(1) FROM table
-- UPDATE table SET col='val'
WHERE 1=1 BEGIN TRANSACTION
UPDATE table SET col='val' WHERE 1=1
ROLLBACKGood decisions come from experience. Experience comes from making bad decisions.
It always does.
> Well, it teaches me to do more diverse tests when doing destructive operations.
Or add some logging and do a dry run and check the results, literally simple prints statements:
Then dump out to a file and spot check it five times before running for real.