News for hackers. Stuff that matters.

A comment on Extracting data from Wikipedia using curl, grep, cut and other bash commands. Parent comment

Re:Extracting data from Wikipedia using curl, grep, cut and ... by Symbiote on Monday August 15, 2016 @04:02PM (#12292976)
> far preferable

Well, that depends. Shell scripts can be commented, and they can be built progressively and interactively by building a pipeline. That's a great choice for a one-off task, and in my experience much faster than many other approaches.

It also works well for tasks close to the system. For example, our users are able to download large archives of data, and we keep over 99% of such downloads indefinitely. We delete downloads > 100GB after they're more than 6 months old.

With a shell script run by cron that's achieved with find + rm + curl|jq (to tell the API the download is deleted).

[ Parent | on HN ]
Re:Extracting data from Wikipedia using curl, grep, cut and ... by minimaxir on Monday August 15, 2016 @05:26PM (#12293594)
To clarify, by data processing I mean extract-transform-load, where the outputs of the extract/transform phase(s) may not be immediately obvious. Bash as an simple automation tool for simple tasks is fine.
[ Parent | on HN ]