Re:Help us collect modern Python NumPy code solutions
by Beltiras on Monday March 28, 2022 @08:36AM
(#30829842)
Fixed some of those. Simple case of just pushing the edit button and sending a PR.
Re:Help us collect modern Python NumPy code solutions
by johndough on Monday March 28, 2022 @09:23AM
(#30830261)
The suggested solution https://github.com/Onelinerhub/onelinerhub/blob/34467e427cc6... still runs out of memory with many files.
I believe the original intention behind using mode="a" was to append to the output file while reading the input files at the same time. This way, there is no need for an ever-growing string array.
But there are still many other issues like using default platform string encoding instead of detecting it properly or at least using utf-8, checking for ".txt" anywhere in the path instead of at the end, and not closing the input files with a context manager like the output file, which suggests that this code is just pierced together from various sources.
A robust solution would require many more lines.
Re:Help us collect modern Python NumPy code solutions
by hermitdev on Monday March 28, 2022 @12:54PM
(#30832609)
For bonus points: how about properly handling UTF-8 byte order marks (BOM), too? The way this code is written if the files contain a BOM, you're going to pass the BOM right into the destination file. Fine for the first file, not so fine for any other file. Dropping a BOM mark in the middle of a file isn't going to result in valid UTF-8...
Re:Help us collect modern Python NumPy code solutions
by mynameismon on Monday March 28, 2022 @10:31AM
(#30830868)
Another memory issue (although unlikely) could also be the file itself being larger than the RAM size, which can be avoided by chunking the textfiles.
Repository claims modern code, but uses Python 2 (end of life January 1, 2020) https://github.com/Onelinerhub/onelinerhub/blob/main/python/... Copied from https://wiki.python.org/moin/Powerful%20Python%20One-Liners
Incorrect indentation causing unnecessarily large memory usage https://github.com/Onelinerhub/onelinerhub/blob/main/python/...
Undefined behavior for i = 31 https://github.com/Onelinerhub/onelinerhub/blob/main/c/bits_...
Printing of uninitialized memory https://github.com/Onelinerhub/onelinerhub/blob/main/c/set_v...
Undefined behavior https://github.com/Onelinerhub/onelinerhub/blob/main/c/swap_...