News for hackers. Stuff that matters.
Re:How I Automated My Job by ggregoire on Friday June 08, 2018 @10:52AM (#17265972)
Almost 10 years ago, when I just started to work, I had to do a repetitive task. Something in IIS about 30-50 subdomain bindings. I told to myself “I’m not going to do that manually, such a waste of time, let’s write a script!”. 5 hours later, I was still working on this script and I was stuck. I asked some help to a senior colleague and his reply was: “you know, it would have taken you only 10 min to do it manually”. :)
Re:How I Automated My Job by dougmwne on Friday June 08, 2018 @11:04AM (#17266117)
You are so right from a business standpoint. But what about personal development? This is one of the ways developers are born. Sure the first one or ten attempts to automate something might be a net loss to the business, but eventually that junior employee has leveled up their programming skills and developed an estimating sense of when to automate and when to suck it up. It's easy to sit at our place of knowledge and tut at the beginers but we were all beginers once.
Re:How I Automated My Job by shostack on Friday June 08, 2018 @03:38PM (#17269004)
I'm an "early" programmer and still poking my way around various toy side projects to learn concepts and such.

Ubuntu and Bash had been somewhat of a mystery to me previously, but I eventually got a virtual dev environment setup and was comfortable getting in with navigating to my Vagrant folder, then executing 'vagrant up', 'vagrant ssh', etc. and then navigating to my project.

That comfort turned to annoyance as I realized how unnecessary those few keystrokes were. And despite how little time they took, I found myself getting angry at the repetition of having to enter them.

So I decided to conquer my Bash scripting fears, and wrote what many would consider to be a trivial script to navigate to the Vagrant folder on my host machine, boot up the VM, and then navigate to the appropriate project folder in the VM. Along the way I learned more about aliases, and also how to pass commands from a host to a guest machine which was one of the roadblocks I encountered.

I now can boot up and get into my project folder with a simple three character alias from within any directory on my host machine.

Again--any experienced developer would be able to script this in minutes most likely. Back then it took hours. And now those are the most satisfying three characters I type, as every time I enter them, I get reminded of the progress I'm making and that even trivial things may be candidates for automation.

Re:How I Automated My Job by sgwae on Saturday June 09, 2018 @02:08AM (#17271783)
You could still save yourself the misery of typing one or two characters if you fix that alias.
Re:How I Automated My Job by shostack on Sunday June 10, 2018 @01:02AM (#17276637)
I know, right? I'm actually leaning in the direction of making it a little less spartan by having it either accept a parameter for a project name to go directly to that folder, or prompt me to enter a number to select from a list of projects gathered from the folder names in my parent project directory.
Re:How I Automated My Job by otisfunkmeyer on Saturday June 09, 2018 @10:51AM (#17273297)
you just blew my mind. im not even kidding.
Re:How I Automated My Job by Latteland on Sunday June 10, 2018 @02:35PM (#17279499)
Doesn't every developer eventually alias building to 'b' or maybe 'm'? It's like a law of nature. Or you can just rebuild automatically every time you save a file, like js people do.
Re:How I Automated My Job by jamesb93 on Friday June 08, 2018 @07:33PM (#17270590)
Just in most cases its not necessarily better to have written the script. You could learn the script in a situation in which the time payoff isnt so terrible.
Re:How I Automated My Job by philipov on Friday June 08, 2018 @09:16PM (#17270972)
How can you possibly estimate the time payoff if you've never done it before?

In any case, I think it's more than just the time payoff. Getting repetitive tasks automated allows you to reallocate that cognitive overhead to other details, and that's a payoff that's hard to quantify, but is critical for leaving your mind free to focus on more important complexity.

Re:How I Automated My Job by dmethvin on Friday June 08, 2018 @11:19PM (#17271330)
Quite often you have to manually do the task at least once to determine how to automate it. That tells you your upper bound on how much time can be saved.

If for example it only took 15 minutes of manual work it's highly unlikely that the payback of automation would occur any time soon. That's especially true because the 2nd or 3rd time there may be different special cases and you'd have to go back and change your automation.

Re:How I Automated My Job by nwatson on Saturday June 09, 2018 @03:19AM (#17271936)
Automation's payoffs aren't necessarily in time saved. Quality can be a more important consideration. "Did I forget a step?" "Did I forget to verify this aspect of the configuration?" "Oh ... am I sure I typed in the new domain name correctly each of the eight times I needed to type it in in the various contexts?"

Automating a task usually means each important parameter is entered only once, that all steps are completed, and that the results are checked.

Re:How I Automated My Job by lmz on Saturday June 09, 2018 @04:47AM (#17272134)
Of course if we agree programming is not so easy, verifying that the script actually works as intended will take roughly the same time. Not to mention thinking about what it should do when faced with an unusual situation.
Re:How I Automated My Job by isostatic on Saturday June 09, 2018 @05:53AM (#17272273)
Automation can still be a benefit even if it initially takes more time. People doing repetitive tasks make mistakes.

Spend 10 hours writing a script to automate 50x10 minute jobs. You spend an extra hour, but you end up with 50 consistent results, rather than 40 successful jobs and 10 with minor mistakes.

Re:How I Automated My Job by z0r on Friday June 08, 2018 @11:38AM (#17266496)
Often the time invested in writing an automation script far outweighs the time spent doing it manually, even when you account for every time the script will ever be run. And it can still be worth it if the script never makes a mistake that human might.
Re:How I Automated My Job by masukomi on Friday June 08, 2018 @01:20PM (#17267639)
even when it doesn't. I find that if you are careful about adhering to the unix philosophy of tiny scripts that do one thing well then you end up with lots of tools to solve problems specific to your problem domain that get reused in future tools.
Re:How I Automated My Job by matthberg on Friday June 08, 2018 @02:26PM (#17268350)
This xkcd is really relevant, being a chart of the time invested vs time saved, not even including the accuracy and learning value: https://xkcd.com/1205/
Re:How I Automated My Job by defertoreptar on Friday June 08, 2018 @05:26PM (#17269835)
My rationale is that every time I automate a task, I get better at automating tasks. Losing a battle here and there helps me win more battles later on, and by the end of my life I will have netted more time saved than lost.

Also, automation work is much more interesting than doing stupid, repetitive tasks.

Re:How I Automated My Job by jrockway on Friday June 08, 2018 @12:08PM (#17266813)
I always look at it as decreasing a dependency on me personally. If I'm sick, the task can't be done at all, for example. So even if you spend an hour automating something that can be done manually in 10 minutes, it's a good use of time because the script doesn't get sick, doesn't go on vacation, doesn't leave for another company, etc.
Re:How I Automated My Job by taneq on Friday June 08, 2018 @10:04PM (#17271112)
The flip side of this is your company gradually becoming dependent on a sprawling and ever-growing tangle of likely undocumented scripts. This can be a time bomb (no matter how good your intentions) waiting to explode if you ever leave.
Re:How I Automated My Job by jrockway on Monday June 11, 2018 @01:53PM (#17286661)
I disagree. If you make a mess, you have a mess... but at least your intentions were at one point clear enough to encode as a computer program. While painful to pick apart during an emergency, it's easier than finding the one person in the world with that knowledge and getting them to explain it to you. Having your business depend on a messy pile of scripts is better than having your business depend on someone who died a few months ago. Having your business depend on clean well-maintained code is even better... but great is the enemy of good.
Re:How I Automated My Job by ivan_ah on Saturday June 09, 2018 @09:00AM (#17272804)
Not necessarily. If the script is documented and written cleanly (e.g Fabric3 which has very little magic, and reads almost like Bash), then in can server as "runnable documentation". In this case automation could be really good for continuity and busfactor.

But yeah... got to put some effort into it. Like all docs, script-docs worn't write themselves.

Re:How I Automated My Job by taneq on Saturday June 09, 2018 @11:27PM (#17276290)
Agree that it's not unavoidable, but the problem is deeper than just the scripts themselves being well documented and straightforward. The main risk (IME, anyway) is awareness / discovery of the scripts in a timely manner - the fact that everything 'just works' means that it can be months or years before something breaks, at which point it may be too late to ask the person who set it up. Hopefully there's good handover from the outgoing staff member to a technically proficient colleague. If so then everything might be OK but if not then a catastrophic failure may well be the first sign that something is wrong.

(A recent example of much simpler automation going horribly wrong was a company losing their top-level domain name because it was set to auto-renew using an ex-employee's personal credit card, and their email address as contact. This worked for years until that card was cancelled and the warning emails bounced off the closed account.)

Re:How I Automated My Job by vkjv on Friday June 08, 2018 @12:58PM (#17267371)
I can think of a few benefits, even if it takes longer: you may need to do something similar again and now you have the script, you've gained experience and you might right the next script faster, correctness.

I think correctness is the most important one. Even if a manual task is quick, ensuring that you do it correctly and exactly the same way for all 30-50 things is extremely tedious. If one out of those 50 subdomains was behaving differently and you needed to investigate exactly why, that could quickly erode any time savings.

Re:How I Automated My Job by jessaustin on Friday June 08, 2018 @04:59PM (#17269639)
Correctness is huge. The other day a colleague and I were figuring out some things about scheduled work days over the next few years. After I had futzed about in a spreadsheet for five minutes, he said "this is ridiculous!" and started furiously paging through a paper calendar. We had our results nearly simultaneously, but he was off by over a month. After that we just used the spreadsheet.
Re:How I Automated My Job by hectormalot on Saturday June 09, 2018 @02:54AM (#17271883)
Correctness also works the other way around. We needed to link to databases with non-matching customer IDs, the analyst on the team spent several hours automating fuzzy matching of customer names in R before we concluded it was easier and more correct to build a mapping table by hand for the few 100 that didn’t match.
Re:How I Automated My Job by maxxxxx on Friday June 08, 2018 @11:07AM (#17266155)
From a short time perspective you wasted your time. But in the long run you learned valuable skills that later may have enabled you to automate other more important things.
Re:How I Automated My Job by sh87 on Sunday June 17, 2018 @03:28PM (#17333871)
I am a script junkie. I have to, just have to script every task out. Here's what I've realized.

First do it manually, then automate it.

Doing manual tasks is frustrating enough to motivate me to automate it. Plus I have a much better understanding of the inputs, the required outputs and constraints for the script to be more useful.

Let your scripts evolve.

Don't force everything in the first go. Simplicity is the key. Let them be rigid, readable and in a single file at first. Then, introduce flexibility as required. Abstract out core functions into individual scripts/functions/modules. This builds a certain sense of familiarity with the code that is just too endearing and eventually very productive.

Reuse your mini-scripts.

Each of my scripts does one thing. There are super-scripts that combine many of those for a task that I am performing at the moment. DRY principle. The more simple, atomic and mutually agreeable your scripts are the more powerful they get. This is just the unix way of doing things. Nothing new.

Write consistent code comments

You will write. You will forget. Make the code comments personal and consistent enough for you to immediately recognize them.

Use version control

Absolutely use it. There will be many versions of your core scripts/functions that dont agree with each other. My personal preference is a version number (semver) suffix on each file to identify which core script is used by which task script.

Reap the rewards

The rewards of putting in the effort to me have been akin to compound interest on an early investment. I have reused core scripts so heavily that prototyping any new idea is pretty easy. If there is something that a script does not do, I know what I need to read over the next weekend :)

Re:How I Automated My Job by dmortin on Friday June 08, 2018 @11:02AM (#17266102)
If it's a one time task then there is no need for a script, but if you have to do it again and again then it's very much worth it to write a script.
Re:How I Automated My Job by lloeki on Saturday June 09, 2018 @04:50AM (#17272142)
Scripts/automation can be worth it even for one-shot tasks for which time to create a script > time to do the task manually:

a) They help in applying forethought to a process. Maybe you can even dry-run the thing.

b) They help documenting what has been done, and how, which can be invaluable in a team setting.

c) They train you into getting incrementally better at automating, and at sensing when to automate and when not to†. If it's painful, you're not doing enough of it.

† I find that for the ones that would take ages to properly script, more often than not a combination of manual (like preparing a normalised input for the script using vim) and automated subtasks can do wonders, leveraging the best of wetware and software worlds.

Re:How I Automated My Job by scarface74 on Saturday June 09, 2018 @03:02PM (#17274462)
The only thing that I will do manually in any environment is putting sensitive data into configuration. With IAM roles, I can usually avoid that. I will create the roles manually.

If you go by the old mantra of treating servers like cattle and not pets, you have to automate everything. My VMs (EC2 instances) can be destroyed, replicated, auto scaled, etc. at any time. I have to automate no matter how long it takes to figure out.

Re:How I Automated My Job by jillesvangurp on Saturday June 09, 2018 @02:06AM (#17271771)
I make these tradeoffs a lot. Automating is fun but time consuming. Doing things manually is tedious. However, automating has one benefit: you make things easier for the next time you do something similar because you learn something. Usually requests for tedious things are not isolated incidents. So I have a bunch of commandline stuff that I know how to glue together like csvkit, jq, xargs, and a simple bash script that adds functions to do rest against our api server. Using that I can do easy bulk operations against our db without investing in product features, ui, etc.
Re:How I Automated My Job by charlieflowers on Saturday June 09, 2018 @06:01AM (#17272292)
Terrance Parr's motto (the creator of ANTLR) is “Why program by hand in five days what you can spend five years of your life automating?”
Re:How I Automated My Job by TomK32 on Saturday June 09, 2018 @12:33AM (#17271555)
sure, but mess it up once and you spent time to figure the error and fix it.

or go on holidays and your colleagues just gotta run the script.

the big benefits in scripts that is often overlooked: Given the same conditions and input, the output will always meet your expectations. There's no chance for human error.

Re:How I Automated My Job by meej on Friday June 08, 2018 @09:12PM (#17270953)
I am reminded of this Brian Marick classic:

http://www.exampler.com/testing-com/writings/automate.pdf

Re:How I Automated My Job by alifaziz on Saturday June 09, 2018 @05:55AM (#17272280)
Upfront time investment to build automation script sometimes more than manual. But most of the time it's a one time effort to save more of your future time.
Re:How I Automated My Job by agumonkey on Saturday June 09, 2018 @12:34AM (#17271560)
A classic. Someone else might have said: next script will only take 15min.

Maybe partial automation would have been a good roi.

Or maybe being deep into manual is also neat for your brain.

Re:How I Automated My Job by mattkrause on Sunday June 10, 2018 @01:10PM (#17279011)
For data analysis, there’s a lot to be said for examining the data instead of blindly piping your data through something.
Re:How I Automated My Job by ajnin on Friday June 08, 2018 @11:05PM (#17271290)
Re:How I Automated My Job by snerbles on Friday June 08, 2018 @11:09AM (#17266173)
From a pure time savings perspective, this makes sense - naturally, links to XKCD are already popping up.

From a maintenance perspective, a well-written script can serve as a reference after the original task was long forgotten. A poor substitute for documentation, but when performing software archaeology on legacy systems you really need everything you can get.

Re:How I Automated My Job by lbotos on Saturday June 09, 2018 @10:08PM (#17276047)
XKCD always delivers: https://xkcd.com/1205/