Re:Never run ‘python’ in your downloads folder
by heavyset_go on Sunday August 23, 2020 @05:52PM
(#24255584)
You can go a long way with jq, pup and similar tools.
As an exercise, I used Bash to build scraping and download scripts for about a dozen websites. I found that I only needed to dip into Python when doing complicated operations on strings, or I needed access to data structures.
I actually liked the approach of using Bash first for scraping, because it gave me direct access to Bash's superior IPC and concurrency constructs. I could plug in a Python script into the pipeline and it wouldn't be any different than interacting with another Bash process.
As an exercise, I used Bash to build scraping and download scripts for about a dozen websites. I found that I only needed to dip into Python when doing complicated operations on strings, or I needed access to data structures.
I actually liked the approach of using Bash first for scraping, because it gave me direct access to Bash's superior IPC and concurrency constructs. I could plug in a Python script into the pipeline and it wouldn't be any different than interacting with another Bash process.