My thirty years of dodging repetitive work with automation tools

Re:My thirty years of dodging repetitive work with automatio... by nyx on Monday February 07, 2022 @02:47PM (#30249983)

This article is a lengthy preamble to an advertisement for a SaaS product, but I'm thankful you at least get to a conclusion before the evangelism begins: there's a basic pattern here, i.e. "get data from X, transform it, and put it in Y", and the tools, systems, and platforms listed are all solutions to the same basic problem.

I'm sure there's a use case and a market for this product, but I'd wager that I'm not alone among the HN audience in thinking that I'd rather have a couple shell scripts in a Docker container on my own hardware or whatever than spend a bunch of money irreversibly trapping my automations in some proprietary cloud-hosted thing where they can be held ransom come the renewal date.

[ Parent | on HN ]

Re:My thirty years of dodging repetitive work with automatio... by gorjusborg on Monday February 07, 2022 @03:13PM (#30250339)

What saves me the most time is not "automation" but something I call "Don't waste time learning things that are likely to go away".

So yeah, I'll just leverage the skills I've leveraged over and over to make a living.

[ Parent | on HN ]

Re:My thirty years of dodging repetitive work with automatio... by randito on Monday February 07, 2022 @05:18PM (#30252150)

That's a really interesting point.

This seems to explain a skepticism that I have, with regards to new tools, new languages, and especially "a new [x] that will solve all your problems."

[ Parent | on HN ]

Re:My thirty years of dodging repetitive work with automatio... by sidpatil on Monday February 07, 2022 @06:31PM (#30253010)

https://en.m.wikipedia.org/wiki/Lindy_effect

[ Parent | on HN ]

Re:My thirty years of dodging repetitive work with automatio... by Ma8ee on Monday February 07, 2022 @04:41PM (#30251673)

And that basic pattern is usually called Extract, Transform and Load (ETL) and there are numerous implementations. Some respectable Open Source ones too.

[ Parent | on HN ]

Re:My thirty years of dodging repetitive work with automatio... by biellls on Monday February 07, 2022 @03:02PM (#30250201)

I think there's space for an open source library that can help with what you described by using just python and YAML. We originally created https://github.com/typhoon-data-org/typhoon-orchestrator to orchestrate ETL workflows, which would be a superset of the use cases you described. Our next goal is to allow deployment to AWS lambda which can be a good compromise between getting locked in with SAAS and hosting your own infrastructure.

Also check out Zappa's scheduled tasks that have a similar goal and inspired our library. We used it initially as a backbone and ran into a series of issues that forced us to write our own version of it, but depending on your goals it could be enough.

[ Parent | on HN ]