Re:Papermill: Parameterizing, executing, and analyzing Jupyt...
by fastasucan on Friday September 20, 2024 @04:10PM
(#41605172)
> I can't believe I'm getting down voted for suggesting one shouldn't build complex workflows using notebooks.
I think it is more the way you express your general attitude and how you look down upon your colleague just because they have found notebooks perfectly suited for their work.
Perhaps the largest critique against notebooks is that they don't enforce a linear execution of cells. Every data scientist I know has been bitten by this at least once (not realizing they're in a stale cell that should have been updated).
Sure you could solve this by automating the entire notebook ensuring top-down execution order but then why in the world are you using a notebook like this? There is no case I can think of where this would be remotely better than just pulling out the code into shared libraries.
I've worked on a wide range of data science teams in my career and by far the most productive ones are the ones that have large shared libraries and have a process in place for getting code out of notebooks and into a proper production pipeline.
Normally I'm the person defending notebooks since there's a growing number of people who outright don't want to see them used ever. But they do have their place, as notebooks. I can't believe I'm getting down voted for suggesting one shouldn't build complex workflows using notebooks.