The hard part is that introducing a tool like git (which requires you to choose moments to take a snapshot of the file, and then add some commit message) breaks the flow of interactive experimentation that notebooks are so good for. And then we need to find a way to make those commits useful, because the time ordering of commits could be different from the time order in which cells were run! That is what is crucial to making computations reproducible — viewers should be able to replay the history of how a notebook result came to be. (EDIT: Note that this is the case only for stateful computations -- if a notebook interface was used to construct a dataflow graph (like spreadsheets) with values updating live, then this wouldn't be so much of a problem. More fundamentally, it is not at all obvious that thinking of notebook contents as akin to code is the best way to use version control)

I wonder whether there is a solution along the lines of auto-committing each cell before it’s executed and the results just after the cell is executed. Otherwise a user has to do too much manual organizing, which is a problem the notebook should ideally solve. When a user is happy with the experiments and the provenance of their results, they should be able to use an interactive rebase to create a cleaner version to share/archive.

I'm not a Jupyter user, but I solve the reproducibility problem with Make.

As a project moves from exploration toward production, the entire thing is wrapped into a Makefile that can flow from raw data to publication in a single call to make.

To have reproducible prototypes, I use Make to wrap the whole workflow in docker. Then I push the code to gist and forget about it. Although GitHub gist doesn't allow binary file, images embedded in .ipynb (JSON), on the other hand works in gist. Here is an example.

https://gist.github.com/ontouchstart/854a3c280b81f530d3ae9cb...

The notebook generated by nbconvert (see the instruction in the Makefile) is too big to display in GitHub gist Web UI but works fine in nbviewer.

https://nbviewer.jupyter.org/gist/ontouchstart/854a3c280b81f...

This has been my solution as well. There’s little that feels as good as running `make -B report` and watching the whole thing be rebuilt from scratch.

How do you manage encapsulating each step, and passing data between them?

I don't do it well. See

https://github.com/4kbt/ReplicableAnalysis

and

https://github.com/4kbt/PlateWash

as examples. The former is smaller/less complicated. The latter was my thesis work -- more complicated and (unfortunately) abuses recursive calls to Make.

Would that be easier with rake than with make?

This is a super valuable perspective! We (https://qri.io) are building a kind of git/github for datasets and are hoping to talk to would-be users about just this issue. Would love to have your feedback on it (particularly on how commits are registered). Mind if I ping you at the email address you listed? - Rico

Tighter integration with git is very interesting, but this is sadly just integration with github.

I think coupling to github makes sense if you are a building a dev-support service, but for a end user it makes little sense to wed the vcs to a specific website.

The RCloud project covered some of this ground https://cscheid.net/2015/08/17/collaborative-visual-analysis... It takes the view that everything should be saved and versioned. In hindsight it seems obvious that this can overwhelm people with dead ends and scratch work and in general the flat workbook space doesn't provide enough help with organizing results. There are some other ideas mentioned in the conclusion of the RCloud paper.

RStudio’s Markdown notebooks do not suffer from this and save a separate output file that can be gitignored.

And they pay for this on other accounts:

No inline rendering of markdown.

Opening an .Rmd file is a lottery to see if rendered graphs and tables still exists.

Tables render completely differently in editor, HTML and pdf

For me, markdown is meant to be readable even when not rendered. I could see how not having persistent graphs and tables might be an issue, but my own philosophy is to start fresh each time - I treat it like a templating language with some convenient rendering features for prototyping, rather than like an IDE.

Your last point also has an upside - it's using different engines (Rmarkdown vs. Sweave). I can write whatever HTML or LaTeX code I want, depending on what's appropriate. I wouldn't want to have to make web documents with LaTeX, nor would I want to make PDFs with HTML.

> No inline rendering of markdown.

That's incorrect, take a look here -> https://blog.rstudio.com/2016/10/05/r-notebooks

RMarkdown and Knitr are dramatic improvements in terms of final outputs and VC relative to notebooks. Notebook believers (Satan worshippers, imho) would suggest that notebooks are best for developing in and not primarily made for use as final outputs.

I have used jupytext (https://github.com/mwouts/jupytext) for this and it seems to work great - it outputs a separate .py file which is easily diff-able.

Thanks for the note; this looks good.

I think they fundamentally json is just the wrong format for these files. Speaking from (ancient and limited) experience I made a little notebook-style interpreter for learning scala back in 2009 or so called scalide. It saved its files ("scalapads") to XML. XML actually worked better in some ways since most of the code could live between the tags unescaped (sans < > &) so it merged / diffed the user code well. The meta-level stuff (cell boundaries etc) needed by the notebook ... not so much.

In json the code has to be escaped into strings, and json is really finicky about syntax (e.g. no trailing commas). So it doesn't work well.

I never got the chance to redo it, however the solution I was leaning to for my post "I won the lottery, I can work on fun stuff" attempt was to store the meta-code in a version of the host language(s), with some simple syntax that could live comfortably in the comments of various different languages to do things like encode the cell divisions and so on.

Basically something like: #notebook[lang=python]

#cell[lang=python] def add(x,y): return x + y #endcell

//notebook[lang=scala]

//cell[lang=python] def add(x: Int, y: Int) = x + y //endcell

This I think would be beneficial for a couple of reasons.

1. Better diffing / merging.

2. One click toggle between show source and view as notebook mode, which would really allow this to work in an IDE like vscode pretty seamlessly. The cells become something akin to //#regions in the IDE. But at the end of the day you are still editing a source code file, so you can edit the whole file easily.

3. The keyboard shortcuts for executing and jumping between cells would generally work in raw code mode, so you could just edit there continuously and manually writing out //cell //endcell. Also the executing results could appear in block comments inline in the editor, off to the side, or in a popup above, the code you are editing.

4. The IDEs could uprender the comment-syntax into cells as they gained better support for the paradigm (similar to how they do for code folding / syntax higlighting already).

5. Eventually, perhaps a cross language, metasyntax could be established to make things a bit more concrete than magic comments (get ready for some serious bikeshed painting though!)

The closest I have seen anything come in this regard is Quokka however it's not quite all the way there.

The closest thing I've seen to what you described would be... Emacs. It actually uses the "metadata in file-specific comments" paradigm. You can put file-local values for Emacs variables in comments at the top or bottom of your file, like described in [0].

Your example could be rewritten as:

  # -*- notebook-lang: python -*-

or

  // -*- notebook-lang: scala -*-

Still, the usual way of using Emacs for "interactive notebooks" is via org-mode, which is a better Markdown with support for (among other things) executing code blocks straight in the org document you're writing. This way, Emacs support all your points 1 to 5, and is generally more powerful than Jupyter or other similar things, but it also means you can kiss any kind of collaboration goodbye.

For some weird reason, the more powerful a tool, the less likely it is other people will be using it.

--

[0] - https://www.gnu.org/software/emacs/manual/html_node/emacs/Sp...

Well, the reason is not so weird. Generally speaking, the more powerful the tool, the higher the bar. More time and effort is required to learn it and become proficient with it. When it comes to the very powerful tools, few will have the aptitude or be prepared to put in the effort to learn them.

For those who don't, less powerful tools take their place and proliferate.

As you say, emacs checks all the boxes but the majority is not prepared to learn it and prefers to program throught their browser.

Along the same lines, I would love to see a syntax something like this:

mynotebook.py

    ### (cell boundary)
    """Top-level unused strings (docstring-esque) rendered as markdown"""
    def add(x, y):
      return x + y
    # jupyter-output-hash: 0123abc (which would link to some external key-value storage for the project)

Anything in something other than the primary language could be in something like `execute_scala(""" scala code """)` - which would execute properly given proper globals.

As long as the output-hash storage is treated as append-only and is highly available (output cells could even be encrypted for security if this was a public cloud service, or you could even use a local or shared filesystem), then this file would not only parse and run as a perfectly valid Python file, but it would also hold references to outputs in a source-control friendly way. IDEs could show the cell outputs inline. If you rerun your notebook and get different outputs for some reason, `git diff` tells you exactly where things changed without being too messy. Basically, put outputs in off-chain storage, and just be a literate code file.

I feel like this would address most people's needs, no?

I nicknamed one we used during the Ebola epidemic (tight deadlines, lots of people working, etc.) "The Wall of Madness".

There were tons of

## JOHN: DONT RUN PAST HERE, EVERYTHING BROKEN

comments.

I’m glad I’m not the only one. When I inherited some “production notebooks” (if that’s a thing) I couldn’t believe it was nearly impossible to do basic things such as test and review changes (via version control).

At our company if it's in a notebook it's not considered ready for production, it must run as a script before being considered for Eng to take over from DS. It's actually not that hard to write a notebook in such a way that it converts easily to a script. Just check and make sure that your variables/functions/whatever are initialized above the cell(s) they're used in, declare all imports in the top cell, and periodically move cells to fix any inconsistencies with these rules (checking that you didn't break anything of course). I've always said that Data Scientist doesn't mean, "I don't do engineering," good basic eng practice helps make more productive data science and brings it into production more robustly. How do you know your models work well if the code that generated them is inscrutable?

I wonder how much of the "3 engineers for 1 data scientist" ratio I hear all the time is due to Data Engineering being assigned the role of cleanup to code that should be better in the first place.

I think cleanup is part of it. I also have noticed as a guy on the DS job family, but who has taken a large interest in SDE work, separating the job families can result in churn. For example, I might think of three model choices A, B and C. C may be the worst of the three, but only very marginally worse. It can also be the case that C is an order of magnitude easier to keep and maintain in production.

I've seen cases where the wrong choice here ends up requiring three SDEs for half a year, where if they gave up a tiny benefit of the best model, they could have done it with 1 SDE in 1 month.

You don't use Jupyter notebooks in production; they are super useful for pitching ideas to clients/bosses and doing some early prototyping. I feel sorry for anyone that has to work with "pure data scientists" that have no clue about software engineering practices...

FWIW, Netflix uses Jupyter notebooks in production, using nteract UI:

https://medium.com/netflix-techblog/notebook-innovation-591e...

https://nteract.io/

This approach seems promising, particularly as it facilitates cross-disciplinary collaboration.

It depends on what you're doing, yeah? In RMarkdown notebooks... yeah, I wouldn't write models in one. But if the focus is on embedding some visualizations and tables into a document, and then refreshing the document to every so often pull in new data, I can see that as a production use for a notebook. TL;DR: Can be useful for reporting, wouldn't use it anywhere else in the pipeline.

"Production notebooks" should not be a thing... Unless prefaced by a bold RUN ALL.

I'm coming to realize one of the key skills for a data engineer to have nowadays is "productionizing" notebook code from data scientists and PMs and teaching them to make it more testable and modular in the first place.

Though the name "data engineer" may be newish, the role is really an old one - and this aspect has always been the single most important part of the role.

If you are interested in workbooks which are collaborative and versioned, take a look at http://datalore.io/

Version control is transparent and integrated and it's possible to work with workbooks collaboratively.

Any chance of this being offered for on-prem install in the future? Looks interesting but cloud only makes it a no go for my team.

We're building something just like that at qri (https://qri.io) a free and open source dataset version control system. Right now all datasets on qri are public by default, but we're working toward supporting. encryption and private networks.

We are seriously considering such a possibility. Do you have any specific requirements for on prem installation?

Basically just the ability to run on Linux.

Jupytext linked elsewher eont he thread seems like a step in the right direction. Instead of changing the whole tool, accept that you're always going to be married to github and change the serialization-layer to be source control friendly. Basically, split the input from the output+metadata and flatten it all to text. Then you can source control it fine and if you need use the output+metadata fold them back in.

Diffing JSON as text must be painful. Diffing JSON as data should be somewhat simple.

Are there any merge tools that offer features for this a lot more sophisticated than basic text comparison?

Not merge, but jq can diff, and it can also do a consistent dump (jq -cS) for you.

Edit: to clarify, jq -S does deep keys sorting.

  $ echo '{"z":{"b": "second", "a": "first"}, "x": 4, "y": 7}' | jq -S
  {
    "x": 4,
    "y": 7,
    "z": {
      "a": "first",
      "b": "second"
    }
  }

Powershell has Compare-Object, which will diff .NET objects. It has the convenient alias "diff". JSON can be converted to .NET objects by ConvertFrom-JSON.

So you can import 2 JSON files and diff them in Powershell.

I'm a huge fan of databricks, its got github sync built into it