Notebooks are not much better than copy-pasting from a notepad or editor into an interpreter. They’re great for reports, but dangerous for presenting the illusion of reproducibility.

At best you’re constantly restarting your kernel and clearing output. More likely, output from cell #7 has modified output [138] but you haven’t updated the chart produced in cell #17 (or some similar craziness). Not much better than programming with GOTOs.

“But they’re great for reporting and visualization!” you might say. If you’re building any report of value, though, it will influence important decisions. That’s the reason your code shouldn't live in a notebook. It should be in a library, covered by unit tests, so that those decisions aren’t based on faulty logic.

Notebooks (and this command-line ebook) assume that the input data is static (i.e. an ad-hoc analysis) which is a more typical use case.

Dynamic data/reporting is a different thing entirely, at which point things like business intelligence software and dashboards come into play, and outside the scope of a command line anyways.

I might have needed to edit down my comment, but you locked on to the least important part of my argument.

If reproducibility is important, like you say, then a notebook is the last thing you want. Your code needs to be tested and designed like the software it is, instead of tossed into some notebook that does not fit into classic software practices.

Notebooks however let you run code blocks in arbitrary orders, delete ones but keep their results in memory, change them without rerunning and change code with none of the downstream dependencies updating.

It's possible (actually very easy) to have code which works as you're making it but not if you run it from scratch.

Which is why notebooks typically a) declare all imports/dependencies in the first block and b) the entire notebook is rerun from a fresh session before publishing (and notebooks have a keyboard command for doing that too).

In all my years of work with Notebooks, I've never had an issue with downstream dependencies.

Important distinction: the notebooks themselves don't do this. The notebook authors, by convention, are supposed to do this.

This is no different than engineers who are, by convention, supposed to write bug-free code. Even with this convention, however, devs still rely on testing to decrease the likelihood of bugs.

No similar tooling exists for notebooks, which is why I recommend moving re-used logic into a tested codebase.

> ... using an IDE/Notebook.

I was with you until the end. The important part is that a script is clear and commented, and can be written to fail informatively. These are benefits of using a scripting language.

Who cares if an IDE was used rather than a traditional editor?

Dynamic notebooks and the JSON mess they generate are a personal peeve of mine, and IME, the enemy of reproducibility.

If it’s for a one-off analysis, but processing is complicated enough that scripts don’t make dependencies between processing steps clear, use GNU Make.

If it’s a data product that’s running in the background, consider something like Airflow.

Still, I find coreutils and friends incredibly useful for interactively sizing up text data.

I very frequently do data tasks as bash command lines and do so within an org-mode code block. So, at least with org-mode, notebook computing and data processing in the shell are not mutually exclusive.

EDIT: Also I should note that notebooks are not the only (or in my opinion best) way to present a reproducible analysis.

I'm working on my data science degree (previously a senior sysadmin), and this is also the workflow I have settled on. Along with versioning/diffs so you can walk back in time to see your changes in a script, I think it's one of the more robust and reproduceable systems around.

So my scripts focus on the data, but around the code block are comments and notes about what is being done, etc.

I've recently been looking at how to integrate makefile into this since watching the following video: https://www.youtube.com/watch?v=sd0HhW8vkSQ

All that said, kudos to the author for this, reading through it I see a few things I hadn't even though of yet, and I live on the command line in general. If it werent for browsing the internet (eww in emacs is nice though) and gaming, I don't think I'd even need a desktop environment.

> If it werent for browsing the internet (eww in emacs is nice though) and gaming, I don't think I'd even need a desktop environment.

I switched to EXWM[1]; Firefox looks and acts like an Emacs buffer.

[1] https://github.com/ch11ng/exwm

Code clarity includes the interfaces of the tool of choice. I suspect that shell files checked in to a version control system will have a longer working life because IDE/Notebook interfaces are more likely to change and become incompatible over time driven by the commercial needs of their vendors.

It seems to me that encapsulating analyses in bash scripts would help with reproducibility.

scripts marshalled with makefiles have helped me with long period recurrent tasks that also slowly evolve

Is there a tool that would create a notebook from the command-line?

I think that depends on what you mean; Jupyter lets you run magic[0] commands (run bash commands in subprocesses), and there's also a bash kernel[1].

[0]: https://blog.dominodatalab.com/lesser-known-ways-of-using-no...

[1]: https://github.com/takluyver/bash_kernel

Jupyter notebook has Bash kernels

You can also do the same in RMarkdown, if you're more of an R user

notebooks are garbage because of the arbitrary order of execution

The cells have numbers corresponding to execution, and you can restart and rerun the entire notebook, or down to a certain cell very easily.

jupyter specifically discourage this by hiding the button to run the whole notebook in a menu and by not assigning a shortcut to the functionality. It's better in colab though, where at least there is a button.

The thing is that people who are not professional programmers often don't realize that this is a danger, and their thing doesn't work and they don't know that it's because they're in a weird state emanating from the random execution of code blocks. So, I mean, notebooks are useful and cool, but they're definitely dangerous, especially for people who aren't software engineers, which is likely a huge fraction of their users