News for hackers. Stuff that matters.
Re:DTrace on Windows by otterley on Monday March 11, 2019 @02:16PM (#19361323)
NTFS is in some ways arguably still more advanced than ZFS: think reparse points, alternate streams, transactions, BitLocker integration, and shrinkable volumes, to name a few.

Also, NTFS already has snapshots via Volume Shadow Copy.

ZFS is great but it’s not going to be a viable replacement as it exists today.

Re:DTrace on Windows by psykotic on Monday March 11, 2019 @07:43PM (#19364018)
> think reparse points, alternate streams, transactions, BitLocker integration, and shrinkable volumes, to name a few.

NTFS's USN change journal (which is separate from the transaction journal) is also useful as a robust offline alternative to ReadDirectoryChangesW file monitoring. Toy example: https://gist.github.com/pervognsen/bcc610d6a5ae6cbc3b2b4f7aa....

Anyway, the problem I've always had when using non-standard file system metadata like alternate streams or sparse files is that they interoperate poorly in practice. There's lots of code and programs out there that wants to treat everything like a plain old file and hence won't preserve alternate streams or sparseness. From my recollections this is true even for several of Windows's built-in utilities. Same issue if you want to transfer files over a socket or pipe or a non-NTFS storage medium like a FAT-formatted USB drive or a NetApp filer or a source control system. As a result this kind of thing is only really useful when deployed in a bubble; I believe WSL uses alternate streams for Unix permission bits and other Unix-specific file properties. Even though those WSL guest files live in a directory tree on a normal NTFS volume, they strongly caution you against touching them directly outside of WSL, presumably for the aforementioned reasons.

Aside from features, NTFS has some performance issues compared to other modern file systems, notably with lots of smaller files.

Re:DTrace on Windows by mehrdadn on Tuesday March 12, 2019 @01:22AM (#19365623)
Some minor notes:

- For permissions WSL uses EAs; for capabilities it uses ADS

- I'm not sure if it's really NTFS that is slow with lots of small files or the I/O subsystem in general... I got the impression it's the latter but not sure.

Re:DTrace on Windows by setquk on Monday March 11, 2019 @02:19PM (#19361351)
Not a lot of this matters in the real world. The real shame is the shitty small file performance due to MFT contention on NTFS.
Re:DTrace on Windows by otterley on Monday March 11, 2019 @02:21PM (#19361370)
It absolutely matters: a fair amount of software, both enterprise and consumer, already uses these features. Malware detection, for example, makes heavy use of reparse points.
Re:DTrace on Windows by blattimwind on Monday March 11, 2019 @03:06PM (#19361821)
Windows Update and Windows Installer use NTFS transactions.
Re:DTrace on Windows by setquk on Monday March 11, 2019 @03:35PM (#19362060)
From Microsoft's documentation:

Microsoft strongly recommends developers utilize alternative means to achieve your application s needs. Many scenarios that TxF was developed for can be achieved through simpler and more readily available techniques. Furthermore, TxF may not be available in future versions of Microsoft Windows. For more information, and alternatives to TxF, please see Alternatives to using Transactional NTFS.

https://docs.microsoft.com/en-us/windows/desktop/fileio/abou...

Re:DTrace on Windows by mehrdadn on Monday March 11, 2019 @05:21PM (#19362938)
I'm pretty convinced that documentation is BS. I could be wrong I guess, but I honestly don't see them removing TxF anytime soon. They didn't exactly build their business on breaking backwards-compatibility of APIs.
Re:DTrace on Windows by setquk on Monday March 11, 2019 @06:44PM (#19363608)
They've burned a lot of APIs in the last few years.
Re:DTrace on Windows by mehrdadn on Monday March 11, 2019 @06:53PM (#19363669)
Examples?
Re:DTrace on Windows by setquk on Monday March 11, 2019 @07:28PM (#19363913)
Re:DTrace on Windows by mehrdadn on Monday March 11, 2019 @07:39PM (#19363991)
They're not removed though? They're 'deprecated' as in they don't want you to depend on them, but they're still very much there and usable.
Re:DTrace on Windows by cryptonector on Monday March 11, 2019 @04:25PM (#19362464)
ZFS snapshots work well enough for that.

The Sun Storage Appliance used ZFS snapshots for updates, and it worked quite well.

Re:DTrace on Windows by otterley on Monday March 11, 2019 @06:07PM (#19363343)
Did that change recently? I haven't used Windows since XP, but back then, they used Volume Shadow Copies (VSS) for making backups used for reverting changes, not NTFS transactions.
Re:DTrace on Windows by tjalfi on Tuesday March 12, 2019 @10:29AM (#19368439)
Windows Update switched to using NTFS transactions in Windows Vista.
Re:DTrace on Windows by setquk on Monday March 11, 2019 @03:32PM (#19362033)
Reparse points could be emulated at API level.
Re:DTrace on Windows by ygra on Monday March 11, 2019 @02:58PM (#19361772)
Is the MFT even documented and available via APIs on Windows? Could NTFS (at the expense of losing backwards compatibility in the file system (which already only extends to features that have been there before)) just change how the MFT works transparently for well-behaved applications (i.e. those that do not grovel around in undocumented data structures)? Admittedly, there may still be a number of applications that decide to »optimize« a system by digging around in parts that they have reverse engineered, and those might simply corrupt the file system when running on a newer NTFS version, but I guess that problem may have existed previously as well.
Re:DTrace on Windows by setquk on Monday March 11, 2019 @03:29PM (#19362007)
MFT is an internal structure. They could fix it TBH but the issue is that this would require an NTFS filesystem revision and for those of us who suffered through NT4's broken ass filesystems might start complaining :)
Re:DTrace on Windows by chungy on Monday March 11, 2019 @09:07PM (#19364471)
> think reparse points, alternate streams, transactions, BitLocker integration, and shrinkable volumes, to name a few.

All of these except shrinkable volumes (somewhat) are already part of ZFS, also assuming that you replace "BitLocker integration" with more general "encryption integration"

ZFS and NFSv4 were both designed to be able to serve the entire subset of NTFS features over the network, to interop with Windows computers.

Re:DTrace on Windows by otterley on Tuesday March 12, 2019 @01:30AM (#19365659)
Reparse points and ADS? Do you have a reference handy? I’ve never seen such things or anything like them in ZFS.
Re:DTrace on Windows by chungy on Tuesday March 12, 2019 @03:07AM (#19365988)
Reparse points are basically more primitive directory-only symlinks, aren't they? Even the symlinks introduced in Vista are better...

As for ADS: basically xattrs. They can be arbitrarily named and each xattr on ZFS can be up to 16EiB large (same as the primary bin for file content). (As an aside, they aren't fully usable on Linux since Linux itself imposes a 64KiB limit on an xattr's content -- but that's not a ZFS limitation)

Re:DTrace on Windows by otterley on Tuesday March 12, 2019 @11:46AM (#19369287)
> Reparse points are basically more primitive directory-only symlinks, aren't they?

Absolutely not! Reparse points are directories that are associated with drivers that extend the capability of the filesystem itself. They can be used to implement symlink-like behavior, but that's only a single use case that scratches the surface of their potential power.

See https://docs.microsoft.com/en-us/windows/desktop/fileio/repa... for more information.

Re:DTrace on Windows by chungy on Tuesday March 12, 2019 @02:07PM (#19371162)
Thanks, I wasn't aware of that... terms seem to get interchanged in the Windows world often enough.

It doesn't sound like it is really an inherent property of the file system, but just controlled by a driver. I don't see why they couldn't be made to work on any other file system, such as FAT or ZFS.

Re:DTrace on Windows by ygra on Tuesday March 12, 2019 @05:52AM (#19366611)
Reparse points are directory entries that are special and interpreted by the file system driver (or some other higher-level component that's not the file system itself). Directory junctions (which you mean here, I guess) are reparse points, as are symlinks and a bunch of other things.
Re:DTrace on Windows by Gladdyu on Monday March 11, 2019 @02:33PM (#19361507)
Though it might finally do away with the ever-persisting Windows error messages about "this file is already open" followed by a search through all of the processes that might have it open to terminate the offender.
Re:DTrace on Windows by otterley on Monday March 11, 2019 @02:45PM (#19361631)
That error is orthogonal to the filesystem in use. Every OS that provides exclusive locking, regardless of filesystem, will return an error if more than one process wants to acquire such a lock.
Re:DTrace on Windows by tracker1 on Monday March 11, 2019 @04:54PM (#19362708)

    PS> $lockedFile="C:\Windows\System32\wshtcpip.dll"
    PS> Get-Process | foreach{$processVar = $_;$_.Modules | foreach{if($_.FileName -eq $lockedFile){$processVar.Name + " PID:" + $processVar.id}}}
Re:DTrace on Windows by jodrellblank on Monday March 11, 2019 @08:18PM (#19364209)

    PS C:\> gps |? {$_.Modules.FileName -match "wshtcpip\.dll"} | select name, id
Re:DTrace on Windows by felixfbecker on Monday March 11, 2019 @06:04PM (#19363322)
That foreach looks like it should really be a where
Re:DTrace on Windows by tracker1 on Monday March 11, 2019 @07:04PM (#19363751)
It was copied from stackoverflow... apologies, I just knew it was possible.
Re:DTrace on Windows by reitanqild on Monday March 11, 2019 @03:36PM (#19362062)
> Every OS that provides exclusive locking, regardless of filesystem, will return an error if more than one process wants to acquire such a lock.

Obviously.

The super annoying thing is when the OS doesn't try to tell you which process(es) keeps it open and doesn't even ship with built in tooling to let you find out on your own.

I think that is the point GP tries to make.

Re:DTrace on Windows by Negitivefrags on Monday March 11, 2019 @04:01PM (#19362243)
Open up "Resource Monitor" and go to the "Disk" tab. Under "Disk Activity" you can see all open file handles.
Re:DTrace on Windows by kevin_thibedeau on Monday March 11, 2019 @06:10PM (#19363369)
Certainly not a sensible solution like having a dialog button that gives you all the relevant information immediately and empowers you to solve the problem without a goose chase.
Re:DTrace on Windows by coldtea on Monday March 11, 2019 @04:44PM (#19362630)
Which are thousands...
Re:DTrace on Windows by psykotic on Monday March 11, 2019 @09:10PM (#19364480)
You need the Sysinternals utilities to make life tolerable: https://docs.microsoft.com/en-us/sysinternals/downloads/hand... from the command line or Ctrl-F in https://docs.microsoft.com/en-us/sysinternals/downloads/proc... if you prefer a GUI. They do a brute-force search over the open handles in all processes, but it's effective.
Re:DTrace on Windows by noisem4ker on Tuesday March 12, 2019 @08:30PM (#19374526)
I prefer this utility: http://www.emptyloop.com/unlocker/

It lets you see which process locked a file, and optionally remove the lock.

Re:DTrace on Windows by kevingadd on Monday March 11, 2019 @03:12PM (#19361862)
The way filesystem locking works on Windows is an intentional decision and not an NTFS thing. It's there for a good reason even if it's inconvenient. The alternative has some real downsides. "Two CMD windows have the same CWD but are showing different folders" is not a user-friendly experience.
Re:DTrace on Windows by blibble on Monday March 11, 2019 @04:03PM (#19362257)
I bet pretty much every Windows user would trade confusing command prompts in strange circumstances for not having to reboot for pretty much every single Windows Update (due to it being unable to update files that are in use)
Re:DTrace on Windows by muststopmyths on Monday March 11, 2019 @05:46PM (#19363155)
It would be horrendous to even try to troubleshoot a system with a dozen different version of OS DLLs loaded because the system had not been rebooted for a dozen patches.

Or, imagine every copy of Word you have running is using a different set of binaries.

No, thank you.

Re:DTrace on Windows by kevin_thibedeau on Monday March 11, 2019 @06:12PM (#19363385)
And yet Linux distros some how manage years of uptime with continuous updates.
Re:DTrace on Windows by muststopmyths on Monday March 11, 2019 @05:27PM (#19362992)
download sysinternals tools, one of which is "handle" for quickly finding open files.

Yeah, it should come with the OS, but that's Windows for you.

Re:DTrace on Windows by rkagerer on Monday March 11, 2019 @03:20PM (#19361941)
Send To > Unlocker
Re:DTrace on Windows by voltagex_ on Monday March 11, 2019 @06:02PM (#19363295)
Hasn't worked since XP, has it? I'd love to see the source for that - I don't know how they were hooking the Copy dialog.
Re:DTrace on Windows by rkagerer on Thursday March 14, 2019 @02:07AM (#19386853)
http://www.emptyloop.com/unlocker/

I use it in Win 7; I think it works most of the time (not always).

Re:DTrace on Windows by usefulcat on Tuesday March 12, 2019 @12:15AM (#19365375)
If a file is damaged by bit rot, NTFS will happily serve up garbage. So those are nice features, but viewed in that light they're little more than window dressing.
Re:DTrace on Windows by Demiurge on Tuesday March 12, 2019 @09:09AM (#19367728)
You say that as if bit rot is the black plague of data storage. In the past 20 years I have encountered 'bit rot' with NTFS... not once. I have encountered bad blocks and replaced drives once in a while by monitoring SMART status, which probably prevented the rot. It is definitely great to have protection against it, but to call other features window dressing when others have used them more often than they thought of checksumming feature is not objectively fair.
Re:DTrace on Windows by usefulcat on Tuesday March 12, 2019 @09:38AM (#19367975)
In the past 20 years I've lost multiple files to bit rot on NTFS. In my opinion, the primary purpose of a filesystem is to reliably store data. Checksumming is not particularly complex technically, does not impose significant run time costs, and allows the filesystem to at least detect when data has been damaged. Unless there's some other very compelling reason to not do checksumming, it's reasonable to consider it's omission to be a de facto bug.
Re:DTrace on Windows by Demiurge on Tuesday March 12, 2019 @12:02PM (#19369528)
Does this mean ext4 is bugged too? We many filesystems with this bug, and we lived with this bug for a while. Anyway, I don't disagree on the technical cost/benefit analysis, but still, unfair.
Re:DTrace on Windows by kuwze on Monday March 11, 2019 @03:30PM (#19362009)
Didn’t know any of that. Thanks!