In Jupyter I approach this by structuring my exploratory analysis in sections, with the minimum of variables reused between sections.

Typically the time-intensive data prep stage is section 1.

The remaining sections are designed essentially like function blocks: data inputs listed in the first cell and data outputs/visualizations towards the end.

Once I decide the exploratory analysis in a section is more-or-less right, I bundle up the code cells into a standalone function, ready for reuse later in my analysis.

Jupyter notebooks can easily get disorganised with out-of-order state. However that is their strength too: exploratory analysis and trying different code approaches is inherently a creative rather than a linear activity.

Maybe I'm missing a joke here, but if that's your workflow then there's absolutely no advantage to notebooks over something like Spyder or even VS Code.

No, that's not the workflow. You work in the notebook as normal but from time to time (say every two hours) rerun the whole thing.

One advantage of this is that it forces you to name your variables such that they don't overwrite each other. Further down the line this enables sophisticated comparisons of states (e.g. dataframes) before and after (something data scientists need)

If you have a few long data loading and preprocessing steps it's a pain to wait for them to run again, people try to avoid it.

When something odd begins to happen, they don't immediately consider the possibility that it's not their bug and waste time trying to 'debug' the problem instead of just rerunning the notebook.

Would it be a solution to store intermediate computations to an in-memory or disk database like Redis, SQLite? It is a matter of few minutes to run a docker instance and write simple read / write + serialize Python util functions?

Surely, it would be a solution, but I don't think for an average data scientist it's a matter of few minutes.

You don't reload every time you write a line of code. Nobody's insane like that. You reload every two hours or so. This is good enough for most except most extreme data sets.

Well if block 1 takes ages but everything after that is dependant 2->3->4 etc. Obviously it would be nice to just re-run block two and have those changes cascade

I break long running data imports out into seperate Notebooks or .py files, and persist the results.

Always restart&re-run for usable results.

That’s what I’d always do. On more complex notebooks, though, is it possible that isn’t a solution? I wouldn’t think so but I am happy to be surprised. Then again I use notebooks only at the end of a project to present work in “executable presentation” style. Restart and Rerun all has been always been sufficient for me. More generally, I took a look at notebooks, thought, “Why develop with all the extra baggage” and left it at that until ready to experiment with presentation methods for (tight) core ideas.

Why are you even using notebooks at all then?

> One advantage of this is that it forces you to name your variables such that they don't overwrite each other. Further down the line this enables sophisticated comparisons of states (e.g. dataframes) before and after (something data scientists need)

Also, not sure about you, but I like seeing all of my outputs on a single browser page without having to write any glue code whatsoever.