Re:Copying data is wasteful, mutating data is dangerous
by masklinn on Thursday January 09, 2020 @02:05PM
(#22004325)
The problem with the library doing this is you end up with the same issue as in Haskell: you accumulate thunks and both memory use and location of CPU hotspots becomes much harder to predict and troubleshoot.
Re:Copying data is wasteful, mutating data is dangerous
by zzzeek on Thursday January 09, 2020 @02:23PM
(#22004540)
so...turn off the "thunking" feature while you are debugging.
Re:Copying data is wasteful, mutating data is dangerous
by JeremyBanks on Thursday January 09, 2020 @02:57PM
(#22004961)
That won't help when you're debugging the performance implications of thunking.
Re:Copying data is wasteful, mutating data is dangerous
by zzzeek on Thursday January 09, 2020 @03:52PM
(#22005577)
compare memory / callcounts / time spent with thunking turned on vs. off would give a good idea of it.
i can say for my side of things thunking is a super huge performance gain as it allows for quick gathering of expression intent and then the computation side on the other side of a cache.
I was doing something similar for SQLAlchemy recently. SQLAlchemy is all about capturing Python operators and operands into data structures that represent intent which can be invoked later.
edit: here is a demonstration: https://gist.github.com/zzzeek/caa4a7ed94f326fbbc031acecb9d7...
edit edit: it is actually possible with numpy by using the Dask library: https://dask.org/