Re:Data Science at the Command Line
by confounded on Saturday January 27, 2018 @07:16PM
(#16248896)
> ... using an IDE/Notebook.
I was with you until the end. The important part is that a script is clear and commented, and can be written to fail informatively. These are benefits of using a scripting language.
Who cares if an IDE was used rather than a traditional editor?
Dynamic notebooks and the JSON mess they generate are a personal peeve of mine, and IME, the enemy of reproducibility.
If it’s for a one-off analysis, but processing is complicated enough that scripts don’t make dependencies between processing steps clear, use GNU Make.
If it’s a data product that’s running in the background, consider something like Airflow.
Still, I find coreutils and friends incredibly useful for interactively sizing up text data.
I was with you until the end. The important part is that a script is clear and commented, and can be written to fail informatively. These are benefits of using a scripting language.
Who cares if an IDE was used rather than a traditional editor?
Dynamic notebooks and the JSON mess they generate are a personal peeve of mine, and IME, the enemy of reproducibility.
If it’s for a one-off analysis, but processing is complicated enough that scripts don’t make dependencies between processing steps clear, use GNU Make.
If it’s a data product that’s running in the background, consider something like Airflow.
Still, I find coreutils and friends incredibly useful for interactively sizing up text data.