Re:Don't spend 6mins doing it by hand when you can spend 6h ...
by AtlasBarfed on Thursday December 09, 2021 @06:11PM
(#29504377)
I've been using a custom-baked "cluster access" toolkit for automating things for Cassandra (and now other clusters). It takes a while for the playback to happen timewise, but it does start to accumulate, especially as the number of machines increases.
But it becomes a massive load of configuration, and I've had to rewrite from scratch three times.
When you have even just preprod and prod, it can be hard to get the savings, but once you have infrastructure copies in four or five different environments and five or six AWS regions, then things start to really pay off.
My "tool" tries to abstract the means of forming a set of infrastructure, the means for delivering commands to it (ssh, AWS ssm, kubectl, dockerrun, or other means), tries to abstract the clouds to do multicloud (but I've never tested this). Man is it hard to do all that. Config config config, context context context.
But it pays off eventually. Someday I might get the time to open source it, probably when the current job kicks me to the curb and no one has a use for it but me. I won't try to say it would be useful to others though.
That's the kicker on complex automation, with Conway's law in the mix, it is very hard to see what hidden dependcies/hardcodes there are based on your stack, software, team/company, policies, etc. To abstract that is such a mound of configuration it almost starts to collapse upon itself.
It took me three years before I could fully automate a cassandra cluster in multiple regions, data loads, spawn loadtests, run them, and tear it all down in near pushbutton. But man do you get more powerful when you can script stuff like this. Woe be to them that has to try to parse them apart.
I've taken to writing a sort of "workflow engine" that is based on filesystems and subdirectories, so I organize the code into individual scripts I can run together, or I can run it on a parent directory and go "run all these scripts as a workflow". So now I almost have an adhoc workflow engine, which is probably a "any sufficiently advanced code has a bad LISP engine in it" joke almost.
The issue is no one makes a general thing like this I can see. Ansible is tied up in ssh, saltstack relies on their UDP delivery. Most things like those two have bolt-on for k8s but it is hokey.
I do think the "COMMAND LINE" is the most universal standard of automation, but I think it's a bit underappreciated.
But it becomes a massive load of configuration, and I've had to rewrite from scratch three times.
When you have even just preprod and prod, it can be hard to get the savings, but once you have infrastructure copies in four or five different environments and five or six AWS regions, then things start to really pay off.
My "tool" tries to abstract the means of forming a set of infrastructure, the means for delivering commands to it (ssh, AWS ssm, kubectl, dockerrun, or other means), tries to abstract the clouds to do multicloud (but I've never tested this). Man is it hard to do all that. Config config config, context context context.
But it pays off eventually. Someday I might get the time to open source it, probably when the current job kicks me to the curb and no one has a use for it but me. I won't try to say it would be useful to others though.
That's the kicker on complex automation, with Conway's law in the mix, it is very hard to see what hidden dependcies/hardcodes there are based on your stack, software, team/company, policies, etc. To abstract that is such a mound of configuration it almost starts to collapse upon itself.
It took me three years before I could fully automate a cassandra cluster in multiple regions, data loads, spawn loadtests, run them, and tear it all down in near pushbutton. But man do you get more powerful when you can script stuff like this. Woe be to them that has to try to parse them apart.
I've taken to writing a sort of "workflow engine" that is based on filesystems and subdirectories, so I organize the code into individual scripts I can run together, or I can run it on a parent directory and go "run all these scripts as a workflow". So now I almost have an adhoc workflow engine, which is probably a "any sufficiently advanced code has a bad LISP engine in it" joke almost.
The issue is no one makes a general thing like this I can see. Ansible is tied up in ssh, saltstack relies on their UDP delivery. Most things like those two have bolt-on for k8s but it is hokey.
I do think the "COMMAND LINE" is the most universal standard of automation, but I think it's a bit underappreciated.