News for hackers. Stuff that matters.
Re:Windows file access performance compared to Linux by giancarlostoro on Saturday December 29, 2018 @10:41AM (#18784050)
> The NT file system API is designed around handles, not paths. Almost any operation requires opening the file first, which can be expensive. Even things that on the Win32 level seem to be a single call (e.g. DeleteFile) actually open and close the file under the hood. One of our biggest performance optimizations for DrvFs which we did several releases ago was the introduction of a new API that allows us to query file information without having to open it first.

Ouch that sounds painful... Is this why deleting gigs worth of files takes a bit? I could of sworn it's not a huge difference on Linux, at least when using the GUI, maybe when doing straight rm it's quicker.

Re:Windows file access performance compared to Linux by magicalhippo on Saturday December 29, 2018 @11:01AM (#18784158)
Reminds me of a performance optimization I did at work. A previous developer had implemented a routine to clean up old files. The routine would enumerate all the files in a directory, and then ask for the file age, and if old enough delete the file.

The problem was that asking the file age given a path caused an open/close, as GetFileSizeEx expects a handle.

Now, at least on Windows, enumerating a directory gets[1] you not just the filename, but a record containing filename, attributes, size, creation and access timestamps. So all I had to do was simply merge the directory enumeration and age checking.

The result was several orders of magnitudes faster, especially if the directory was on a network share.

[1]: https://docs.microsoft.com/en-us/windows/desktop/api/minwinb...

Re:Windows file access performance compared to Linux by tom_ on Saturday December 29, 2018 @11:49AM (#18784396)
POSIX stuff is terrible for this sort of thing, but luckily it's often a straightforward fix once you know to look for it. Here's a similar fix for the old Win32 version of the silver searcher: https://github.com/kjk/the_silver_searcher/pull/7

Looks like the active Windows port (https://github.com/k-takata/the_silver_searcher-win32) doesn't have this change, but it's possible Mingw32 or whatever exposes this flag itself now (as it probably should have done originally).

Re:Windows file access performance compared to Linux by dataflow on Saturday December 29, 2018 @03:28PM (#18785508)
Path-based I/O seems quite dangerous to me. If everything was path-based, you'd easily have inherent race conditions. You want to delete a directory? You stat() all the files, they look empty, so you delete them... but in between, another process writes to some file (or maybe the user forgets the file is being deleted and saves to something there), and suddenly you've deleted data you didn't expect. When you do things in a handle-based fashion, you know you're always referring to the same file (and can lock it to prevent updates, etc.), even if files are being moved around.

However, to answer your question of why removing a directory is slow... if you mean it's slow inside Explorer, a lot of it is shell-level processing (shell hooks etc.), not the file I/O itself. Especially if they're slow hooks -- e.g. if you have TortoiseGit with its cache enabled, for example, it can easily slow down deletions by a factor of 100x. But regarding the file I/O part, if it's really that slow at all, I think it's partly because the user-level API is path-based (because that's what people find easier to work with), whereas the system calls are mostly handle-based (because that's the more robust thing, as explained above... though it can also be faster, since a lot of work is already done for the handle and doesn't need to be re-performed on every access), so merely traversing the directory a/b/c requires opening and closing a, then opening and closing a/b, then opening and closing a/b/c, but even opening a/b/c requires internally processing a and b again, since they may no longer be the same things as before... this is O(n^2) in the directory depth. If you reduce it to O(n) by using the system calls and providing parent directory handles directly (NtOpenFile() with OBJECT_ATTRIBUTES->RootDirectory) then I think it should be faster and more robust.

Re:Windows file access performance compared to Linux by yason on Saturday December 29, 2018 @04:14PM (#18785717)
You stat() all the files, they look empty, so you delete them... but in between, another process writes to some file (or maybe the user forgets the file is being deleted and saves to something there), and suddenly you've deleted data you didn't expect

This is fundamentally not any different between the systems, race conditions can happen either way. The user could write data to file right before the deletion recurses to the same directory and the handle-based deletion happens. Similarly the newly written data would be wiped out unintentionally.

For files for which access from different processes must be controlled explicitly there is locking. No filesystem or VFS is going to protect you from accidentally deleting stuff you're still using in another context.

Re:Windows file access performance compared to Linux by mehrdadn on Saturday December 29, 2018 @04:36PM (#18785849)
> [...] The user could write data to file right before the deletion recurses to the same directory and the handle-based deletion happens. Similarly the newly written data would be wiped out unintentionally. [...] No filesystem or VFS is going to protect you from accidentally deleting stuff you're still using in another context.

...what? No file system is going to protect you from accidentally deleting in-use files? But that's exactly what Windows does: it prevent you from deleting in-use files. That's what everyone here has been complaining about. File sharing modes let you lock files to make sure they're not written to (and/or read from) before being deleted, it very much need not be the case that the user could write to a file before it's deleted.

Re:Windows file access performance compared to Linux by yason on Monday December 31, 2018 @04:54AM (#18793869)
Read my comment again.

There is an inherent race condition if one program is using a file and another program is deleting it without caring about whether the file is being accessed by other programs.

At that point, all bets are off there regardless of whether the files are accessed by paths or handles.

Windows protects the file from deletion at the exact same time as it is being accessed but does not protect the file from being deleted after it has been accessed. In wall-clock statistics the latter is the way more likely scenario.

So, if an editor saves a document to disk, and another program then deletes the document the editor will happily exit without saving it again thinking that it hasn't been changed.

It doesn't particularly matter whether the two programs clash exactly at the time of saving/deletion or not. The problem exists in the lack of information between the programs and no file system is indeed going to protect you from that.

Re:Windows file access performance compared to Linux by mehrdadn on Monday December 31, 2018 @05:09AM (#18793903)
> So, if an editor saves a document to disk, and another program then deletes the document the editor will happily exit without saving it again thinking that it hasn't been changed.

I'm trying to explain to you that your understanding of Windows is wrong and that this is impossible. As long as that editor has the document open, unless it has explicitly specified FILE_SHARE_WRITE and FILE_SHARE_DELETE, Windows will not allow another program to alter that file. So the editor would very rightly assume upon exiting that nobody has touched that file while it's had that file open.

Re:Windows file access performance compared to Linux by yason on Tuesday January 01, 2019 @04:41AM (#18799566)
I know how opening a file affects the exclusivity of access but commonly applications don't seem to keep the file continuously open except during reading or writing. Maybe some Microsoft applications use that pattern extensively but it generally works to save a file in one program and then open it in another program without closing the file in the first program or quitting it.

Nevertheless, this is getting off topic as the thread started with the question of path vs handle access. I still don't see much value in this exclusivity in the latter case because if you have a conflict to begin with it's just a matter of Murphy's for that conflict to manifest in actually losing data.

Even if your editor keeps the file open for the whole day, and you have this second program that is on the trajectory to delete the file it will eventually get to it at a time when the file is not open and thus not protected by the guarantee of exclusive access.

Re:Windows file access performance compared to Linux by int_19h on Saturday December 29, 2018 @07:50PM (#18786648)
> This is fundamentally not any different between the systems, race conditions can happen either way. The user could write data to file right before the deletion recurses to the same directory and the handle-based deletion happens.

When you hold a handle to a file or directory, you get to decide on the degree of shared access with any other users for the duration that handle is held (FILE_SHARE_*). So this does solve the concurrency problem, by allowing you to, effectively hold a lock on the file until you're done.

Re:Windows file access performance compared to Linux by mirimir on Saturday December 29, 2018 @11:43AM (#18784368)
> I could of sworn it's not a huge difference on Linux, at least when using the GUI, maybe when doing straight rm it's quicker.

Using Linux file management GUIs can be a disaster. Some years ago, I was massaging many GiB of text, using a crude hack, with scripts that did grep, sed, awk, etc. And I had many working directories with >10^4 tiny files each (maybe 10^5 even).

In terminal, everything was fine. But opening a GUI would swap out so much that the box would freeze. I don't know why. I just learned to avoid doing it.

Re:Windows file access performance compared to Linux by giancarlostoro on Saturday December 29, 2018 @12:16PM (#18784566)
Heh, that's fine, I don't mind the terminal or the GUI, as long as you're rationally using either and aren't blaming either for your problems. I see a lot of "I only use the terminal" type of people who screw up GUIs and the reverse can be said, GUI users who screw up the terminal, it's all about RTFM'ing sometimes, or just reading what's right in front of you.
Re:Windows file access performance compared to Linux by mirimir on Saturday December 29, 2018 @01:20PM (#18784898)
Re:Windows file access performance compared to Linux by aasasd on Saturday December 29, 2018 @06:19PM (#18786294)
Not only it's extremely surprising that there's no getting metadata without opening each file―and since DrvFs is only the WSL layer, apparently the system itself still to this day doesn't have such a feature.

But I'm now additionally baffled by how Total Commander managed to feel hella snappy ten years ago while navigating lots of dirs, whereas on MacOS all double-panel managers are meh, not least due to the rather slow navigation.

Re:Windows file access performance compared to Linux by valarauca1 on Saturday December 29, 2018 @06:48PM (#18786402)
10 years ago there was less file system integration, user land virus scanning, kernel level virus scanning, os-hooks, OS-compatibility re-direction, and 32/64bit compatibility checks.

This was mostly added during NT6.0 era, which occured ~12 years ago. VISTA was the first OS using NT6.0 and VISTA was VERY much not in vogue ~12 years ago. In fact it was avoided like a plague as of 2008 (unless you were using 64bit, and had >=4GiB of RAM)

So many were using Windows-XP 32bit, or the NT5.2 kernel. Even those with >=4GiB of RAM were on Windows-XP 64bit, as VISTA had a ton of driver problems.

NT6.0 didn't catch until Windows7 and NT6.1