News for hackers. Stuff that matters.
Re:Help us collect modern Python NumPy code solutions by johndough on Monday March 28, 2022 @09:23AM (#30830261)
The suggested solution https://github.com/Onelinerhub/onelinerhub/blob/34467e427cc6... still runs out of memory with many files.

I believe the original intention behind using mode="a" was to append to the output file while reading the input files at the same time. This way, there is no need for an ever-growing string array.

But there are still many other issues like using default platform string encoding instead of detecting it properly or at least using utf-8, checking for ".txt" anywhere in the path instead of at the end, and not closing the input files with a context manager like the output file, which suggests that this code is just pierced together from various sources.

A robust solution would require many more lines.

Re:Help us collect modern Python NumPy code solutions by hermitdev on Monday March 28, 2022 @12:54PM (#30832609)
For bonus points: how about properly handling UTF-8 byte order marks (BOM), too? The way this code is written if the files contain a BOM, you're going to pass the BOM right into the destination file. Fine for the first file, not so fine for any other file. Dropping a BOM mark in the middle of a file isn't going to result in valid UTF-8...
Re:Help us collect modern Python NumPy code solutions by mynameismon on Monday March 28, 2022 @10:31AM (#30830868)
Another memory issue (although unlikely) could also be the file itself being larger than the RAM size, which can be avoided by chunking the textfiles.