News for hackers. Stuff that matters.
Re:63 Cores Blocked by Seven Instructions by ncmncm on Monday October 21, 2019 @08:13AM (#21311316)
The cause is obvious: they were building on Microsoft Windows, using the NTFS filesystem. Even Microsoft doesn't try to build on NTFS.

Changing any single detail gives better results. Use a Samba share from a Linux filesystem. Run Mingw on a Linux system. Run MSVS in Wine on a Linux system.

Windows is an execution environment for applications. There is no need for, and no value in, actually performing builds in your target execution environment. Use a system designed from the ground up for builds.

Re:63 Cores Blocked by Seven Instructions by brucedawson on Monday October 21, 2019 @11:39AM (#21313520)
> Even Microsoft doesn't try to build on NTFS.

Citation needed. I haven't worked at Microsoft for a while but when I did we built on Windows using NTFS. When I found a correctness bug in NTFS last year I was told that it had been affecting Windows builds, which means they were using NTFS as recently as February 2018.

https://randomascii.wordpress.com/2018/02/25/compiler-bug-li...

The biggest problem I usually encounter with building on Windows is slow process creation.

Re:63 Cores Blocked by Seven Instructions by pjc50 on Monday October 21, 2019 @11:33AM (#21313449)
I don't think this deserves the downvotes it's receiving, but:

> no need for, and no value in, actually performing builds in your target execution environment.

.. is completely antithetical to using Visual Studio, where the convenience of building, running and debugging on the desktop is very handy.

NTFS is horribly slow for certain file operations though. Giant batch deletes can wedge the UI while the system catches up. I have in the past benefited from putting %TEMP% on a RAMdisk, although this is a pain to set up.

I'd love to see Microsoft building a VFS driver for, say, ext2/3. It's not impossible, all the APIs are there to add plugin filesystems to Windows, and the OSS-friendly Microsoft shouldn't have any objection in principle to linking against the GPL'd kernel implementation..

Re:63 Cores Blocked by Seven Instructions by monocasa on Monday October 21, 2019 @01:00PM (#21314553)
A lot of the issues people see isn't NTFS per se, it's how the VFS is connected to the IFS drivers and cache manager. Essientially on windows it goes:

  1) user space
  2) cache manager
  3) filesystem driver
  4) disk device driver
whereas on sane platforms its

  1) userspace
  2) filesystem driver
  3) cache manager
  4) disk device driver
So on windows you as the filesystem have to cache your own metadata reads because only you know when they're invalidated, and the caching layer above you is more focused on data plane. On other platforms with the caching layer below you, you can just pretend you're hitting disk for any accesses (data or metadata), and it's probably already cached.

Just throwing ext2/3 in there wouldn't help because as I said above, it's mainly the system architecture that's screwy.

Re:63 Cores Blocked by Seven Instructions by snagglegaggle on Monday October 21, 2019 @11:50AM (#21313630)
There is WinExt2[0], and also WinBTRFS and WinMD. I've not been able to test them myself. Microsoft briefly allowed Windows Pro users to create ReFS volumes (can still read), but since reneged on that, and I can find no Linux implementation. It looks like WinBTRFS gets usable performance.

[0] https://sourceforge.net/projects/ext2fsd/

Re:63 Cores Blocked by Seven Instructions by youdontknowtho on Monday October 21, 2019 @08:45AM (#21311613)
That's the first I've heard that MS doesn't try to build Windows on Windows with NTFS.
Re:63 Cores Blocked by Seven Instructions by dijit on Monday October 21, 2019 @09:04AM (#21311777)
I don't have first hand experience, but I know some people at MS who work on Xbox (which is a modified version of Windows+HyperV underneath);

From what I understood from them, they do not use NTFS (they use SMB from a clustered filesystem) for builders, but they _do_ use a heavily modified version of windows, incidentally that modified version went on to become "windows nano". What the actual "Windows" team does is a mystery to me though, I would assume it was similar or the same.

Re:63 Cores Blocked by Seven Instructions by youdontknowtho on Monday October 21, 2019 @10:17AM (#21312504)
That's super interesting. I wonder if they have moved to use SMBv3?

I really liked the direction with Nano in 2016, but I guess it makes more sense as a container OS. Still, the latest version is what a lot of people wish they could start an operating system with. NT kernel, no wmi, no servicing, no activation.

Re:63 Cores Blocked by Seven Instructions by pitterpatter on Monday October 21, 2019 @06:43PM (#21318367)
All the Windows devs I've met built locally on their machines for their own day-to-day dev work and testing. I don't know anyone that used a filesystem other than NTFS. But definitely fast NVMe SSDs and disabling Defender for the local repo checkout.

The build machines are a different story and I don't know the specifics of.

Re:63 Cores Blocked by Seven Instructions by dooglius on Monday October 21, 2019 @11:07AM (#21313127)
As a developer, you do some builds on your local box, so it's really up to you what filesystem you use.