Re:Command-line Tools can be 235x Faster than your Hadoop Cl...
by yung_steezy on Tuesday March 08, 2022 @02:58PM
(#30605889)
Bash is a godsend for quick debugging and I can see the temptation to start writing production code using bash. It basically boils down to a few things IMO:
- large bash scripts are hard to read/maintain
- complex modelling chains need intermediary points in the processing
On the latter point I can't count the amount of times where being able to query an athena database has saved a lot of headaches. The overhead from parquet and AWS bills pales in comparison. I'm sure almost everyone already agrees with me here but it's a classic case of the whole being more than the sum of its parts.
Re:Command-line Tools can be 235x Faster than your Hadoop Cl...
by randomswede on Wednesday March 09, 2022 @07:21AM
(#30613670)
And bash is really painful once you're trying to do clever things with structured data.
Excellent glue, but there's also a skill in knowing when you should port your increasingly complicated shell script to another language.
Relevant: "bashML: Why Spark when you can Bash?" (https://rev.ng/blog/bashml/post.html), aka how to deduplicate git repositories using `comm` + `awk`.