News for hackers. Stuff that matters.
Re:How to Grow Software Architecture out of Jupyter Notebooks by leethargo on Wednesday December 12, 2018 @04:31AM (#18662531)
Does not sound odd, but in your case you will store results on disk, rather than in memory? That means that there might be some overhead in (de-)serialization and also some restrictions on what kind of data can be stored as an intermediate result?
Re:How to Grow Software Architecture out of Jupyter Notebooks by AboutTheWhisles on Wednesday December 12, 2018 @11:17AM (#18665128)
You can mitigate the reading from disk by memory mapping the file.

I don't think serialization is a problem either. It is possible to make data structures that always exist as a single span of memory, so that they don't have a separate serialized representation. Most data structures are better served like this anyway, since prefetching and linear memory access is a huge speedup over pointer chasing. Building a data structure out of many separate heap allocations will also slow it down.

Re:How to Grow Software Architecture out of Jupyter Notebooks by gnulinux on Wednesday December 12, 2018 @01:28PM (#18666366)
You can memory map and store binary, in-memory data (as long as you don't use pointers). So there'll be some overhead serializing (removing pointers) but ime it's not noticeable since most serialization tasks can be done in milliseconds (even if the dataset is Terabytes) whereas most algorithmic tasks are in the order of seconds, minutes or hours.
Re:How to Grow Software Architecture out of Jupyter Notebooks by jacobush on Wednesday December 12, 2018 @08:07AM (#18663527)
Probably but it's not a given. It depends on the tools. The tools might work with memory mapped data structures directly.