News for hackers. Stuff that matters.
Re:Git and Jupyter Notebooks Guide by pletnes on Friday July 07, 2023 @08:09AM (#36630476)
Sometimes I work on software development, and this mindset («the only valuable asset is the code») makes total sense. But if I work on analytics / datascience projects, the analysis including outputs could be time consuming to run, validate, and visualize. In these cases, it might be required to version the outputs.

I’ve never used jupyter for taking notes in a lab setting, but with more and more instruments being computer/network connected, I imagine this would make total sense - put your data and notes with your analytics work.

Many jupyter users are not «software developers», they just use code to perform their work.

Re:Git and Jupyter Notebooks Guide by Zandikar on Friday July 07, 2023 @08:51AM (#36630933)
Precisely this. When your output is something like research data, or even just something that generally takes a long human time to complete (hours vs Microseconds) it makes a lot of sense to version and keep outputs, at least on major "versions".
Re:Git and Jupyter Notebooks Guide by adament on Friday July 07, 2023 @01:03PM (#36634771)
But would you version it by storing it as output in an ipnyb file where it is overwritten if you rerun that cell? I would store the data in a versioned database or as separate data files in the repository (possibly stored in git-lfs). And I would store results of the analysis as data files / image files / whatever else, NOT as ephemereal outputs in an ipynb file. But I am pretty far down the “ipynb files are for local use only” path.
Re:Git and Jupyter Notebooks Guide by tnecniv on Friday July 07, 2023 @05:48PM (#36638746)
Yeah if your analysis to takes hours to run, you should really split up the number crunching code and result analysis / visualization. Not only does it make version control of the code easier, you can save the output in an organized labeled manner (time-stamped, etc.) and, if you lose power or the kernel crashes, you don’t need to rerun the lengthy analysis if you want to make a change further down the pipeline.
Re:Git and Jupyter Notebooks Guide by Zandikar on Monday July 10, 2023 @06:16PM (#36673761)
It was an extreme example to drive home the point that one is "human scale time" and one is "computer scale time", people are reading far far too much into my choice of hours specifically there.
Re:Git and Jupyter Notebooks Guide by CardenB on Friday July 07, 2023 @12:18PM (#36633999)
Simply write the outputs to a formatted file and keep it separately?
Re:Git and Jupyter Notebooks Guide by aqsalose on Friday July 07, 2023 @01:44PM (#36635448)
However, I wouldn't then use version control software like Git for versioning analysis objects, as it is designed for text file source control and diffs.

(How one does a diff of a data object look like? If there is a natural text format to save it in, it still is usually quite messy, and Git doesn't really like Gb sized csvs.)

My preferred workflow is to version the source files in Git and store the associated data objects in a separate archive directory with meaningful name and the hash of commit of generating code as metadata attribute.

Now if you had a version control "IDE" software that would render changes in figures and other blobs nicely, then it would make sense to build a workflow around it.

Re:Git and Jupyter Notebooks Guide by tnecniv on Friday July 07, 2023 @05:58PM (#36638826)
What’s your solution for hosting the archive of the data if you want to share it? That’s a weakness in my workflow.
Re:Git and Jupyter Notebooks Guide by chthonicdaemon on Saturday July 08, 2023 @04:22AM (#36642462)
I use nbdime and have integrated it into my git diff so I see graphical diffs. Github renders notebooks with their output intact. They recently announced [rich notebook diffs](https://github.blog/changelog/2023-03-01-feature-preview-ric...) as well.