Re:What’s wrong with computational notebooks?
by roblabla on Monday January 27, 2020 @07:31PM
(#22165296)
Assuming the data isn't changed, thanks to CoW forking wouldn't cause any extra memory usage. If only a subset of data is changed, same thing - only the changed cells will take extra space. The problem only occurs when the whole variable changes - in which case yeah, you're SOL. I wonder what the usage patterns are for such datasets?
[ Parent
| on HN ]
Re:What’s wrong with computational notebooks?
by ivalm on Monday January 27, 2020 @07:36PM
(#22165314)
Personal experience: when first looking at the data I often do lots of map /reduce style operations which might transform large portions of the dataframe.
Question, if you use CoW then presumably your variable blocks are no longer contiguous, wouldn't this really slow down vector operations?
Re:What’s wrong with computational notebooks?
by roblabla on Thursday January 30, 2020 @03:04PM
(#22194927)
> Question, if you use CoW then presumably your variable blocks are no longer contiguous, wouldn't this really slow down vector operations?
I don't think so. Vector operations require the data to be aligned to whatever the vector size is, no? E.G. 16-byte vector ops require the data to be aligned to 16-byte, etc... At least that's my understanding.