News for hackers. Stuff that matters.
Re:NTFS read-write GPL kernel driver by formerly_proven on Sunday August 16, 2020 @05:47AM (#24176507)
os.scandir in Python was designed to encapsulate "listdir replacement APIs" and uses FindFirstFile/FindNextFile on Windows, and also allows most stuff to be done without an extra per-file stat() syscall on Linux. On top of that, when accessing many/all files in a folder in Linux, this gives you the inode, and if you sort your accesses by inode, throughput can rise to the ninth power on hard drives.

On the other hand, if you look at e.g. Windows Explorer. Explorer will take somewhere from a few seconds to ~tens of seconds to list the contents of a SSD-backed directory with a couple thousand files. The same operation on Linux is practically instant. Copying a bunch of small files on Windows takes forever, maybe doing around a few hundred files per second. Meanwhile Linux will happily copy thousands of files per second, all on the same hardware.

Explorer might be using incorrect ways to do I/O, but if that is the case, then that would mean that the proprietary, only-ever-developed-for-Windows-by-Microsoft tool that is also the main way users do I/O doesn't do I/O properly. What would that say about Windows?

Re:NTFS read-write GPL kernel driver by wongarsu on Sunday August 16, 2020 @09:46AM (#24177719)
If I remember correctly Windows Explorer still doesn't support long paths (paths with a total length over 255 characters). It only recently gained the ability to create files starting with a dot. It has no support whatsoever for alternate data streams, let alone more obscure NTFS features.

I wouldn't be surprised if it used the wrong API (in fact the lack of long path support is an example of using an outdated API). The only reason Windows Explorer appears reasonable is because people generally consider creating files you can't access with Windows Explorer a bug or misfeature.

Re:NTFS read-write GPL kernel driver by p_l on Sunday August 16, 2020 @04:50PM (#24181193)
Explorer not only appears to be heavy on the "Windows Classic" APIs, somehow to this day, but it also does a bunch of stuff ower file compared to let's say ls.

In fact, if you use Total Commander, in the settings you will find option regarding things like "getting correct icons for files", with the option for "display icons for all files" explicitly marked as "SLOW".

Explorer always gets the icons, and possibly other metadata.

Re:NTFS read-write GPL kernel driver by bleepblorp on Sunday August 16, 2020 @04:08PM (#24180798)
Os.scandir looks like a very good improvement to Python. I haven't coded anything to Python 3.5 or newer, however, so I have no experience with it.

The last time I needed something along these lines the only option were os.listdir or using FindFirstFile through win32py. Os.listdir was an order of magnitude slower for my use case.