News for hackers. Stuff that matters.
Re:How to Grow Software Architecture out of Jupyter Notebooks by thaw13579 on Wednesday December 12, 2018 @03:38AM (#18662320)
It might seem odd, but I achieve a similar thing by scripting my data processing tasks with GNU Make. You can incrementally add steps to the process, and it will retain the existing results as you go along. My use case is scientific computing, where some of the steps may be the results of a costly optimization or simulation. I can update the workflow quite easily and not worry about redoing the expensive bits or making copies. Of course, all of the dependency management and parallel processing of Make are a bonus as well.
Re:How to Grow Software Architecture out of Jupyter Notebooks by dlcmh on Wednesday December 12, 2018 @04:25AM (#18662503)
Thanks, I’d never thought of doing this before - found 2 resources to help me explore this approach:

http://zmjones.com/make/ http://blog.kaggle.com/2012/10/15/make-for-data-scientists/

Re:How to Grow Software Architecture out of Jupyter Notebooks by leethargo on Wednesday December 12, 2018 @04:31AM (#18662531)
Does not sound odd, but in your case you will store results on disk, rather than in memory? That means that there might be some overhead in (de-)serialization and also some restrictions on what kind of data can be stored as an intermediate result?
Re:How to Grow Software Architecture out of Jupyter Notebooks by AboutTheWhisles on Wednesday December 12, 2018 @11:17AM (#18665128)
You can mitigate the reading from disk by memory mapping the file.

I don't think serialization is a problem either. It is possible to make data structures that always exist as a single span of memory, so that they don't have a separate serialized representation. Most data structures are better served like this anyway, since prefetching and linear memory access is a huge speedup over pointer chasing. Building a data structure out of many separate heap allocations will also slow it down.

Re:How to Grow Software Architecture out of Jupyter Notebooks by gnulinux on Wednesday December 12, 2018 @01:28PM (#18666366)
You can memory map and store binary, in-memory data (as long as you don't use pointers). So there'll be some overhead serializing (removing pointers) but ime it's not noticeable since most serialization tasks can be done in milliseconds (even if the dataset is Terabytes) whereas most algorithmic tasks are in the order of seconds, minutes or hours.
Re:How to Grow Software Architecture out of Jupyter Notebooks by jacobush on Wednesday December 12, 2018 @08:07AM (#18663527)
Probably but it's not a given. It depends on the tools. The tools might work with memory mapped data structures directly.
Re:How to Grow Software Architecture out of Jupyter Notebooks by gnulinux on Wednesday December 12, 2018 @01:25PM (#18666339)
Same! I use GNU Make for this. TBH I've never seen anyone using Make for this and my coworkers think it's a bit odd. But I think it's very practical.