News for hackers. Stuff that matters.

A comment on 1BRC Merykitty's Magic SWAR: 8 Lines of Code Explained in 3k Words. Parent comment

Re:1BRC Merykitty's Magic SWAR: 8 Lines of Code Explained in... by mike_hearn on Sunday March 10, 2024 @04:29PM (#39662306)
Unfortunately there are still cases where local disk I/O can be a serious bottleneck that you do have to be aware of:

1. Cloud VMs sometimes have very slow access even to devices advertised as local disks, depending on which cloud

2. Windows

The latter often catches people out because they develop intuitions about file I/O speed on Linux, which is extremely fast, and then their app runs 10x slower on Windows due to opening file handles being much slower (and sometimes they don't sign their software and virus scanners make things much slower again).

[ Parent | on HN ]
Re:1BRC Merykitty's Magic SWAR: 8 Lines of Code Explained in... by NikkiA on Sunday March 10, 2024 @06:17PM (#39662967)
Apparently, the cause of the long standing windows disk IO problem was discovered a month or so ago, and MS were said to be working on a fix.

Whether it'll be constrained to Win 11 is yet to be seen.

[ Parent | on HN ]
Re:1BRC Merykitty's Magic SWAR: 8 Lines of Code Explained in... by dr_zoidberg on Sunday March 10, 2024 @07:23PM (#39663337)
That's interesting. A project at work is affected by Windows slow open() calls (wrt to Linux/Mac) but we haven't found a strong solution rather than "avoid open() as much as you can".
[ Parent | on HN ]
Re:1BRC Merykitty's Magic SWAR: 8 Lines of Code Explained in... by tomnipotent on Monday March 11, 2024 @08:36AM (#39667403)
It's likely Windows Defender, which blocks on read I/O to scan files. Verify by adding a folder to its exclusion list, though this isn't helpful if it's a desktop app. The difference is most noticeable when you're reading many small files.
[ Parent | on HN ]
Re:1BRC Merykitty's Magic SWAR: 8 Lines of Code Explained in... by mike_hearn on Monday March 11, 2024 @06:08AM (#39666332)
Really? Where did you hear that? Conventional wisdom is based on a post by an MS employee years ago that described the performance issues as architectural.

MS does have something called "dev drive" in Win11. Dev Drive is what ReFS turned into and is an entirely different filesystem that isn't NTFS, which is better optimized for UNIX-style access patterns. The idea is that core Windows remains slow, but developers (the only people who care about file IO performance apparently) can format another partition and store their source/builds there.

[ Parent | on HN ]