News for hackers. Stuff that matters.
Re:Copying data is wasteful, mutating data is dangerous by cultureswitch on Friday January 10, 2020 @07:31AM (#22010477)
This is rather dumb. The only reason there is an intermediate array in the original copy-based code is because the normalization is badly written. After you obtain max and min, normalization is a single map() function, not two.

As for the overall problem: always choose immutability and then make it faster by first taking advantage of optimization patterns that immutability allows you to use safely. In a case like this, doing the normalization lazily is what springs to mind.

If memory allocation and de-allocation is an issue, use more explicit memory management. Though by experience it is rarely the actual problem. There's basically no chance that it is an issue here, Python interpreters aren't stupid enough to give back memory to the OS immediately after a variable falls out of scope.

Re:Copying data is wasteful, mutating data is dangerous by markus92 on Friday January 10, 2020 @07:40AM (#22010524)
Do you have any links to some examples or help on how to do this exactly?

Reason being that in our group we have a ton of normalization code that looks exactly like the 'bad' example from this post. As we use relatively large datasets (4D MRI images), memory is a bit scarce...

Re:Copying data is wasteful, mutating data is dangerous by naveen99 on Friday January 10, 2020 @08:03AM (#22010630)
Pytorch has a built in normalize function: https://pytorch.org/docs/master/_modules/torchvision/transfo... It just subtracts the mean and divides by standard deviation, both in place.