News for hackers. Stuff that matters.
Re:DTrace on Windows by qalmakka on Monday March 11, 2019 @01:59PM (#19361157)
This. They really need a newer, more modern filesystem, especially since Apple has done away with HFS+ and has introduced APFS. ReFS is really dead in the water, and ZFS would give them a powerful, battle-tested filesystem which they could use as a real selling point for Windows Server. Also things such as Windows updates and system restore would benefit a lot from ZFS's native snapshotting capabilities.

Merging DTrace implicitly means they are fine with both the CDDL and Oracle's patents, and thus that ZFS wouldn't be an issue for them.

Re:DTrace on Windows by otterley on Monday March 11, 2019 @02:16PM (#19361323)
NTFS is in some ways arguably still more advanced than ZFS: think reparse points, alternate streams, transactions, BitLocker integration, and shrinkable volumes, to name a few.

Also, NTFS already has snapshots via Volume Shadow Copy.

ZFS is great but it’s not going to be a viable replacement as it exists today.

Re:DTrace on Windows by psykotic on Monday March 11, 2019 @07:43PM (#19364018)
> think reparse points, alternate streams, transactions, BitLocker integration, and shrinkable volumes, to name a few.

NTFS's USN change journal (which is separate from the transaction journal) is also useful as a robust offline alternative to ReadDirectoryChangesW file monitoring. Toy example: https://gist.github.com/pervognsen/bcc610d6a5ae6cbc3b2b4f7aa....

Anyway, the problem I've always had when using non-standard file system metadata like alternate streams or sparse files is that they interoperate poorly in practice. There's lots of code and programs out there that wants to treat everything like a plain old file and hence won't preserve alternate streams or sparseness. From my recollections this is true even for several of Windows's built-in utilities. Same issue if you want to transfer files over a socket or pipe or a non-NTFS storage medium like a FAT-formatted USB drive or a NetApp filer or a source control system. As a result this kind of thing is only really useful when deployed in a bubble; I believe WSL uses alternate streams for Unix permission bits and other Unix-specific file properties. Even though those WSL guest files live in a directory tree on a normal NTFS volume, they strongly caution you against touching them directly outside of WSL, presumably for the aforementioned reasons.

Aside from features, NTFS has some performance issues compared to other modern file systems, notably with lots of smaller files.

Re:DTrace on Windows by mehrdadn on Tuesday March 12, 2019 @01:22AM (#19365623)
Some minor notes:

- For permissions WSL uses EAs; for capabilities it uses ADS

- I'm not sure if it's really NTFS that is slow with lots of small files or the I/O subsystem in general... I got the impression it's the latter but not sure.

Re:DTrace on Windows by setquk on Monday March 11, 2019 @02:19PM (#19361351)
Not a lot of this matters in the real world. The real shame is the shitty small file performance due to MFT contention on NTFS.
Re:DTrace on Windows by otterley on Monday March 11, 2019 @02:21PM (#19361370)
It absolutely matters: a fair amount of software, both enterprise and consumer, already uses these features. Malware detection, for example, makes heavy use of reparse points.
Re:DTrace on Windows by blattimwind on Monday March 11, 2019 @03:06PM (#19361821)
Windows Update and Windows Installer use NTFS transactions.
Re:DTrace on Windows by setquk on Monday March 11, 2019 @03:35PM (#19362060)
From Microsoft's documentation:

Microsoft strongly recommends developers utilize alternative means to achieve your application s needs. Many scenarios that TxF was developed for can be achieved through simpler and more readily available techniques. Furthermore, TxF may not be available in future versions of Microsoft Windows. For more information, and alternatives to TxF, please see Alternatives to using Transactional NTFS.

https://docs.microsoft.com/en-us/windows/desktop/fileio/abou...

Re:DTrace on Windows by mehrdadn on Monday March 11, 2019 @05:21PM (#19362938)
I'm pretty convinced that documentation is BS. I could be wrong I guess, but I honestly don't see them removing TxF anytime soon. They didn't exactly build their business on breaking backwards-compatibility of APIs.
Re:DTrace on Windows by setquk on Monday March 11, 2019 @06:44PM (#19363608)
They've burned a lot of APIs in the last few years.
Re:DTrace on Windows by mehrdadn on Monday March 11, 2019 @06:53PM (#19363669)
Examples?
Re:DTrace on Windows by setquk on Monday March 11, 2019 @07:28PM (#19363913)
Re:DTrace on Windows by mehrdadn on Monday March 11, 2019 @07:39PM (#19363991)
They're not removed though? They're 'deprecated' as in they don't want you to depend on them, but they're still very much there and usable.
Re:DTrace on Windows by cryptonector on Monday March 11, 2019 @04:25PM (#19362464)
ZFS snapshots work well enough for that.

The Sun Storage Appliance used ZFS snapshots for updates, and it worked quite well.

Re:DTrace on Windows by otterley on Monday March 11, 2019 @06:07PM (#19363343)
Did that change recently? I haven't used Windows since XP, but back then, they used Volume Shadow Copies (VSS) for making backups used for reverting changes, not NTFS transactions.
Re:DTrace on Windows by tjalfi on Tuesday March 12, 2019 @10:29AM (#19368439)
Windows Update switched to using NTFS transactions in Windows Vista.
Re:DTrace on Windows by setquk on Monday March 11, 2019 @03:32PM (#19362033)
Reparse points could be emulated at API level.
Re:DTrace on Windows by ygra on Monday March 11, 2019 @02:58PM (#19361772)
Is the MFT even documented and available via APIs on Windows? Could NTFS (at the expense of losing backwards compatibility in the file system (which already only extends to features that have been there before)) just change how the MFT works transparently for well-behaved applications (i.e. those that do not grovel around in undocumented data structures)? Admittedly, there may still be a number of applications that decide to »optimize« a system by digging around in parts that they have reverse engineered, and those might simply corrupt the file system when running on a newer NTFS version, but I guess that problem may have existed previously as well.
Re:DTrace on Windows by setquk on Monday March 11, 2019 @03:29PM (#19362007)
MFT is an internal structure. They could fix it TBH but the issue is that this would require an NTFS filesystem revision and for those of us who suffered through NT4's broken ass filesystems might start complaining :)
Re:DTrace on Windows by chungy on Monday March 11, 2019 @09:07PM (#19364471)
> think reparse points, alternate streams, transactions, BitLocker integration, and shrinkable volumes, to name a few.

All of these except shrinkable volumes (somewhat) are already part of ZFS, also assuming that you replace "BitLocker integration" with more general "encryption integration"

ZFS and NFSv4 were both designed to be able to serve the entire subset of NTFS features over the network, to interop with Windows computers.

Re:DTrace on Windows by otterley on Tuesday March 12, 2019 @01:30AM (#19365659)
Reparse points and ADS? Do you have a reference handy? I’ve never seen such things or anything like them in ZFS.
Re:DTrace on Windows by chungy on Tuesday March 12, 2019 @03:07AM (#19365988)
Reparse points are basically more primitive directory-only symlinks, aren't they? Even the symlinks introduced in Vista are better...

As for ADS: basically xattrs. They can be arbitrarily named and each xattr on ZFS can be up to 16EiB large (same as the primary bin for file content). (As an aside, they aren't fully usable on Linux since Linux itself imposes a 64KiB limit on an xattr's content -- but that's not a ZFS limitation)

Re:DTrace on Windows by otterley on Tuesday March 12, 2019 @11:46AM (#19369287)
> Reparse points are basically more primitive directory-only symlinks, aren't they?

Absolutely not! Reparse points are directories that are associated with drivers that extend the capability of the filesystem itself. They can be used to implement symlink-like behavior, but that's only a single use case that scratches the surface of their potential power.

See https://docs.microsoft.com/en-us/windows/desktop/fileio/repa... for more information.

Re:DTrace on Windows by chungy on Tuesday March 12, 2019 @02:07PM (#19371162)
Thanks, I wasn't aware of that... terms seem to get interchanged in the Windows world often enough.

It doesn't sound like it is really an inherent property of the file system, but just controlled by a driver. I don't see why they couldn't be made to work on any other file system, such as FAT or ZFS.

Re:DTrace on Windows by ygra on Tuesday March 12, 2019 @05:52AM (#19366611)
Reparse points are directory entries that are special and interpreted by the file system driver (or some other higher-level component that's not the file system itself). Directory junctions (which you mean here, I guess) are reparse points, as are symlinks and a bunch of other things.
Re:DTrace on Windows by Gladdyu on Monday March 11, 2019 @02:33PM (#19361507)
Though it might finally do away with the ever-persisting Windows error messages about "this file is already open" followed by a search through all of the processes that might have it open to terminate the offender.
Re:DTrace on Windows by otterley on Monday March 11, 2019 @02:45PM (#19361631)
That error is orthogonal to the filesystem in use. Every OS that provides exclusive locking, regardless of filesystem, will return an error if more than one process wants to acquire such a lock.
Re:DTrace on Windows by tracker1 on Monday March 11, 2019 @04:54PM (#19362708)

    PS> $lockedFile="C:\Windows\System32\wshtcpip.dll"
    PS> Get-Process | foreach{$processVar = $_;$_.Modules | foreach{if($_.FileName -eq $lockedFile){$processVar.Name + " PID:" + $processVar.id}}}
Re:DTrace on Windows by jodrellblank on Monday March 11, 2019 @08:18PM (#19364209)

    PS C:\> gps |? {$_.Modules.FileName -match "wshtcpip\.dll"} | select name, id
Re:DTrace on Windows by felixfbecker on Monday March 11, 2019 @06:04PM (#19363322)
That foreach looks like it should really be a where
Re:DTrace on Windows by tracker1 on Monday March 11, 2019 @07:04PM (#19363751)
It was copied from stackoverflow... apologies, I just knew it was possible.
Re:DTrace on Windows by reitanqild on Monday March 11, 2019 @03:36PM (#19362062)
> Every OS that provides exclusive locking, regardless of filesystem, will return an error if more than one process wants to acquire such a lock.

Obviously.

The super annoying thing is when the OS doesn't try to tell you which process(es) keeps it open and doesn't even ship with built in tooling to let you find out on your own.

I think that is the point GP tries to make.

Re:DTrace on Windows by Negitivefrags on Monday March 11, 2019 @04:01PM (#19362243)
Open up "Resource Monitor" and go to the "Disk" tab. Under "Disk Activity" you can see all open file handles.
Re:DTrace on Windows by kevin_thibedeau on Monday March 11, 2019 @06:10PM (#19363369)
Certainly not a sensible solution like having a dialog button that gives you all the relevant information immediately and empowers you to solve the problem without a goose chase.
Re:DTrace on Windows by coldtea on Monday March 11, 2019 @04:44PM (#19362630)
Which are thousands...
Re:DTrace on Windows by psykotic on Monday March 11, 2019 @09:10PM (#19364480)
You need the Sysinternals utilities to make life tolerable: https://docs.microsoft.com/en-us/sysinternals/downloads/hand... from the command line or Ctrl-F in https://docs.microsoft.com/en-us/sysinternals/downloads/proc... if you prefer a GUI. They do a brute-force search over the open handles in all processes, but it's effective.
Re:DTrace on Windows by noisem4ker on Tuesday March 12, 2019 @08:30PM (#19374526)
I prefer this utility: http://www.emptyloop.com/unlocker/

It lets you see which process locked a file, and optionally remove the lock.

Re:DTrace on Windows by kevingadd on Monday March 11, 2019 @03:12PM (#19361862)
The way filesystem locking works on Windows is an intentional decision and not an NTFS thing. It's there for a good reason even if it's inconvenient. The alternative has some real downsides. "Two CMD windows have the same CWD but are showing different folders" is not a user-friendly experience.
Re:DTrace on Windows by blibble on Monday March 11, 2019 @04:03PM (#19362257)
I bet pretty much every Windows user would trade confusing command prompts in strange circumstances for not having to reboot for pretty much every single Windows Update (due to it being unable to update files that are in use)
Re:DTrace on Windows by muststopmyths on Monday March 11, 2019 @05:46PM (#19363155)
It would be horrendous to even try to troubleshoot a system with a dozen different version of OS DLLs loaded because the system had not been rebooted for a dozen patches.

Or, imagine every copy of Word you have running is using a different set of binaries.

No, thank you.

Re:DTrace on Windows by kevin_thibedeau on Monday March 11, 2019 @06:12PM (#19363385)
And yet Linux distros some how manage years of uptime with continuous updates.
Re:DTrace on Windows by muststopmyths on Monday March 11, 2019 @05:27PM (#19362992)
download sysinternals tools, one of which is "handle" for quickly finding open files.

Yeah, it should come with the OS, but that's Windows for you.

Re:DTrace on Windows by rkagerer on Monday March 11, 2019 @03:20PM (#19361941)
Send To > Unlocker
Re:DTrace on Windows by voltagex_ on Monday March 11, 2019 @06:02PM (#19363295)
Hasn't worked since XP, has it? I'd love to see the source for that - I don't know how they were hooking the Copy dialog.
Re:DTrace on Windows by rkagerer on Thursday March 14, 2019 @02:07AM (#19386853)
http://www.emptyloop.com/unlocker/

I use it in Win 7; I think it works most of the time (not always).

Re:DTrace on Windows by usefulcat on Tuesday March 12, 2019 @12:15AM (#19365375)
If a file is damaged by bit rot, NTFS will happily serve up garbage. So those are nice features, but viewed in that light they're little more than window dressing.
Re:DTrace on Windows by Demiurge on Tuesday March 12, 2019 @09:09AM (#19367728)
You say that as if bit rot is the black plague of data storage. In the past 20 years I have encountered 'bit rot' with NTFS... not once. I have encountered bad blocks and replaced drives once in a while by monitoring SMART status, which probably prevented the rot. It is definitely great to have protection against it, but to call other features window dressing when others have used them more often than they thought of checksumming feature is not objectively fair.
Re:DTrace on Windows by usefulcat on Tuesday March 12, 2019 @09:38AM (#19367975)
In the past 20 years I've lost multiple files to bit rot on NTFS. In my opinion, the primary purpose of a filesystem is to reliably store data. Checksumming is not particularly complex technically, does not impose significant run time costs, and allows the filesystem to at least detect when data has been damaged. Unless there's some other very compelling reason to not do checksumming, it's reasonable to consider it's omission to be a de facto bug.
Re:DTrace on Windows by Demiurge on Tuesday March 12, 2019 @12:02PM (#19369528)
Does this mean ext4 is bugged too? We many filesystems with this bug, and we lived with this bug for a while. Anyway, I don't disagree on the technical cost/benefit analysis, but still, unfair.
Re:DTrace on Windows by kuwze on Monday March 11, 2019 @03:30PM (#19362009)
Didn’t know any of that. Thanks!
Re:DTrace on Windows by vbernat on Monday March 11, 2019 @02:06PM (#19361225)
DTrace has switched to GPL for the kernel part (see https://oss.oracle.com/projects/DTrace/).
Re:DTrace on Windows by hs86 on Monday March 11, 2019 @05:57PM (#19363251)
They imported DTrace from FreeBSD which is still CDDL licensed.
Re:DTrace on Windows by simcop2387 on Monday March 11, 2019 @02:53PM (#19361721)
Neat, I hadn't heard about that before. Wonder how much that will affect adoption in the linux world.
Re:DTrace on Windows by lallysingh on Monday March 11, 2019 @02:52PM (#19361706)
From that page, it looks like dtrace is GPL for the entire linux port.
Re:DTrace on Windows by ezannoni on Monday March 11, 2019 @08:17PM (#19364201)
Hi, Yes, it's GPL (and we got rid of the separate kernel module). The utils portion is UPL.

It's available with new kernels (including 5.0) on github as well.

https://github.com/oracle/dtrace-utils

https://github.com/oracle/libdtrace-ctf

https://github.com/ezannoni/dtrace-linux-kernel

We are working on a Fedora version of the kernel patches.

We are also discussing with the BPF team (on netdev) how to provide DTrace functionality integrated with the Linux Kernel.

Re:DTrace on Windows by fetbaffe on Monday March 11, 2019 @02:08PM (#19361235)
Microsoft has already developt ReFS for this, however I don't know the adaption rate of ReFS on Windows servers.

https://en.wikipedia.org/wiki/ReFS

Re:DTrace on Windows by mixedCase on Monday March 11, 2019 @04:24PM (#19362456)
bcachefs is profiling itself to finally up the Linux file system game. It doesn't look like it will be upstreamed this year, but it appears to be usable and it's being done by someone who really seems to understand file systems and knows of the trade-offs different FS have made and how they played out.
Re:DTrace on Windows by fxfan on Monday March 11, 2019 @03:36PM (#19362067)
Wait- how do you find NTFS underpowered? You make vague claims offering nothing of substance.
Re:DTrace on Windows by qalmakka on Tuesday March 12, 2019 @04:10AM (#19366222)
NTFS holds up quite well when compared with more "traditional" filesystems such as XFS, ext4 and UFS (except the choppy I/O performance, but as someone else said above it might be more of an issue of the subsystems above it); it has much more functionality of almost anyone of those I mentioned above (for instance, I really don't understand why so few filesystems support transparent compression).

I think that filesystems like ZFS and Btrfs are simply one step ahead than anything else, not only in features but also in tooling and UX. Send/receive, CoW snapshots, checksumming are all tremendous features that I think would fit fantastically into the Windows workflow, if Microsoft really managed to integrate them well. For instance, I think a ZFS-based Windows update would be able to do away with transactions and rely instead on datasets to snapshot the system, apply an update and rollback everything if something broke, exposing to power users the tools to clearly understand and rollback their systems themselves by hand.

Obviously all of this can be achieved by rewriting everything from scratch as in house, specific platforms, but I just feel that continuing to replicate in NTFS/ReFS features other open and widely supported filesystems have had for almost a decade is just not the best way forward, and it's not really better for anyone.

Re:DTrace on Windows by magduf on Tuesday March 12, 2019 @01:15PM (#19370540)
>(for instance, I really don't understand why so few filesystems support transparent compression).

Does anyone actually use this on Windows these days? I'm guessing the reason so few filesystems support it now is because the gains aren't worth it for the types of files that take up most of the space on filesystems these days, such as video files. You can't compress x264 video with a general-purpose compression algorithm. Storage space is plentiful and cheap, and (general-purpose) compression yields little gains, and the stuff we're storing now is already compressed (images, video, audio, etc.).

Re:DTrace on Windows by qalmakka on Wednesday March 13, 2019 @08:50AM (#19378060)
I have LZ4 compression enabled on ZFS, and not only it perceivably improves performances on slow drives (due to the lower amount of I/O), it saves actual storage space. An average OS install on hard drive is full of things such as executables and text files which summed up can amount to several gigabytes. These files compress pretty well in my experience, and even if it only amounts to saving a few GBs, it can still be crucial on small SSDs.
Re:DTrace on Windows by Macha on Monday March 11, 2019 @09:06PM (#19364466)
My experience is that NTFS is quite slow compared to ext4 on my machine (both via Windows and via ntfs-3g). Particularly for operations around large numbers of small files like moving/deleting a .git directory.
Re:DTrace on Windows by j16sdiz on Monday March 11, 2019 @09:42PM (#19364622)
Core NTFS is fast.

When you feel slow, it is the abstraction layers and plugable interfaces above it. - and they are API.

Replacing the core won't help the performance. They need to remove some abstraction and break some application.

Re:DTrace on Windows by Sylamore on Tuesday March 12, 2019 @02:19AM (#19365839)
I wouldn't say that ReFS is dead in the water, I just think they shifted the focus to supporting virtualization features first to enable S2D/SDN. ReFS is still being developed and enhanced, the work is mostly in server SKU: https://docs.microsoft.com/en-us/windows-server/storage/refs...
Re:DTrace on Windows by johannes1234321 on Monday March 11, 2019 @09:30PM (#19364584)
> Merging DTrace implicitly means they are fine with both the CDDL and Oracle's patents, and thus that ZFS wouldn't be an issue for them.

Mind that DTrace was ported by Oracle (at least partially) to Oracle Linux first. They haven't done that for ZFS (instead they created bttrfs, before owning Sun)

Re:DTrace on Windows by Neil44 on Monday March 11, 2019 @02:10PM (#19361258)
Adopting ZFS would take away from Storage Spaces, So sadly I don’t see it happening.
Re:DTrace on Windows by qalmakka on Tuesday March 12, 2019 @12:40PM (#19370060)
What's the advantage of storage spaces compared with a software RAID solution such as ZFS's RAIDZ?
Re:DTrace on Windows by delfinom on Monday March 11, 2019 @02:36PM (#19361529)
Storage Spaces would just build around ZFS?
Re:DTrace on Windows by Neil44 on Monday March 11, 2019 @06:46PM (#19363627)
Theres just such duplication of functionality I don't see them bothering.