"But think of the security!" they'll say... of course, it's always a tradeoff. I've experienced a similar problem with a large growing logfile --- appending becomes essentially quadratic, every time the process closes the log the AV opens and scans all of it.
I first noticed it when I would open my downloads folder (because I'm a heathen and never clean it out), and it would sit there and "process" for fifteen minutes. Just clicking realtime protection off fixed it instantly -- but it would seem there's no need to re-scan every file every time a folder is opened, scan them when I click on them, or scan them when I try to open them, or scan them in the background while I still can interact with the folder. Being unresponsive is a cardinal sin for any UX, and that's where Apple shines and MS still continues to drop the ball.
Like I said, I really like Windows 10 -- but it really falls short here.
Frankly I think the entire concept of AV is bankrupt as it always causes many more problems than it is worth. Both false positives and false negatives are common. You shouldn't be afraid of running without Defender because, like all AV, it isn't very good at catching real threats anyway. If you really want to feel safe there are much better ways of doing that, like running untrusted programs in sandboxes or VMs, or using Software Restriction Policies to whitelist what can and cannot run. I haven't played with it at all, but there's even Controlled Folder Access to limit FS rights by application, which isn't a terrible idea given the era of crypto-ransomware.
As you say, CFA is probably the most useful part of Windows Defender. I was very glad to see that included.
I like Windows 10 and generally have been happy with it -- but that particular behavior has been driving me nuts for a while. They really need to fix it.