News for hackers. Stuff that matters.

Ask HN: How do you manage one-off internal scripts?

For a recent project, I've found myself writing quite a few one off scripts for database adjustments, SQL queries for analytics, etc. I've been manually running them myself in the command line, but it's a very messy process. New team members also don't know what scripts are available and where to look. Anyone know of any tools, or have any workflows to better organize these scripts?
( 16 comments | 11 points | on HN )
Re:Ask HN: How do you manage one-off internal scripts? by villasv on Friday August 13, 2021 @10:35PM (#28177316)
If they were truly one-off, you wouldn't need to manage them. Categorize and document. SQL queries for analytics go into something like Metabase. Database migrations go into git. Frequent adhoc scripts go into something like Jenkins so you can trigger them easily. So on and so forth. All of those documented somewhere, like the team's Confluence/Notion/whatever.

But don't just leave them at that, do work on eliminating their need. Sisyphean task but necessary nonetheless.

Re:Ask HN: How do you manage one-off internal scripts? by rolandtshen on Saturday August 14, 2021 @03:33AM (#28178805)
I'm curious about your workflow for reusable scripts with Jenkins. What are the advantages to putting it in Jenkins? How do you set up Jenkins to run these individual scripts?
Re:Ask HN: How do you manage one-off internal scripts? by villasv on Sunday August 15, 2021 @08:39AM (#28188309)
Jenkins is not the ideal place. We take some scripts and create pipelines that just invoke them. Some of these are configured to be scheduled jobs. We have separate declarative pipelines for each one.

But this isn’t good, we have Airflow and that’s where these things should be. Their stay inside Jenkins is technical debt, but at least is much better than not having it automated. The advantage is that you made an ad-hoc process less unstructured, you can add post-failure alarms, you don’t have to worry about long running scripts being interrupted by shabby notebooks, you don’t need to spread production credentials to every developer and so on.

Re:Ask HN: How do you manage one-off internal scripts? by Jugurtha on Monday August 16, 2021 @07:22PM (#28204109)
Add them to the repository and treat them like you would treat other code. Chances are, they'll become some sort of tooling later when people tweak them, improve them, make them modular, generalize them.

We introduce many things that way. First it's just a few commands that do something useful. Then included in the repo. Then tweak. Next thing you know there's an endpoint with that functionality because it's useful and frequently needed.

Re:Ask HN: How do you manage one-off internal scripts? by tcbasche on Saturday August 14, 2021 @01:10AM (#28178167)
The classic is the /scripts directory in your codebase (presumably in VCS). The key is also to trawl through occasionally and remove anything that hasn't been used in a while.
Re:Ask HN: How do you manage one-off internal scripts? by giorgioz on Sunday August 15, 2021 @10:28AM (#28189006)
I do the same. a /scripts folder in the root of the git repo. I keep old scripts mainly to copy-paste the structure and/or pieces for new one-time scripts. It feels a bit dirty but I haven't found a better solution yet.
Re:Ask HN: How do you manage one-off internal scripts? by SirensOfTitan on Saturday August 14, 2021 @08:39PM (#28185233)
If it’s truly one-off (like codemods), I put the scripts in relevant tasks and diffs for reference.

…otherwise we just throw everything into a scripts directory. Keep it flat until it feels messy, then start organizing. All documentation must live with the code, otherwise it gets out of date too easily.

Re:Ask HN: How do you manage one-off internal scripts? by wizwit999 on Friday August 13, 2021 @08:35PM (#28176605)
Funnily, wasn't there just a launch HN yesterday that helps with this: https://news.ycombinator.com/item?id=28156461

Oherwise, I just use git like others said but that can be suboptimal for non technical or semi technical users.

Re:Ask HN: How do you manage one-off internal scripts? by yokoprime on Sunday August 15, 2021 @02:09PM (#28190879)
If it's one-off but has to be worked on for some time, with testing and whatnot it may be placed in a folder for maintenance scripts in the repo. Once it's been executed it can be deleted from source control
Re:Ask HN: How do you manage one-off internal scripts? by marto1 on Sunday August 15, 2021 @03:19AM (#28186845)
I mostly..don't. For one liners, I keep them in a git controlled org-mode file. But for everything else it's all over the place.
Re:Ask HN: How do you manage one-off internal scripts? by fiftyacorn on Sunday August 15, 2021 @01:31PM (#28190571)
I have a repo and raise a jira for each script and record any history or background. Then mark the git history against the jira.
Re:Ask HN: How do you manage one-off internal scripts? by MattGaiser on Friday August 13, 2021 @06:25PM (#28175468)
At a prior company we added them to a folder of misc scripts along with notes on what they were for.

Not the greatest solution but it did ensure they never got lost.

Re:Ask HN: How do you manage one-off internal scripts? by rolandtshen on Friday August 13, 2021 @08:04PM (#28176399)
As more engineers joined and people needed to reuse old scripts, was there some process to discover them on their own? My scripts are also dumped in a repo, but team members are having trouble finding them and running them especially since there are quite a few now.
Re:Ask HN: How do you manage one-off internal scripts? by MattGaiser on Friday August 13, 2021 @08:25PM (#28176540)
A combo of group experience and manually reading them.

Most scripts were never re-used, so it was not a problem all that often.

Re:Ask HN: How do you manage one-off internal scripts? by willcipriano on Friday August 13, 2021 @08:07PM (#28176414)
Find the closest related repo and add a scripts folder has been my approach.
Re:Ask HN: How do you manage one-off internal scripts? by deeteecee on Friday August 13, 2021 @07:26PM (#28176117)
you could just put them in a repo