Change monitoring with inotify is going to hit limits quickly.
I recently wanted to take a look at VSCode and it immediately shat itself over the maximum number of inotify watches being too low (the kernel I'm running restricts this to 8k for non-root users).
I then bumped the limit to the max (512k, I think, or about 7 copies of linux.git)…still no luck. Now imagine you have a Google-scale number of files.
> Especially if you can't or don't want to use memory mapping because it's hard to do well.
Why do new languages never address this issue?
Re:Bazel 1.0
by londons_explore on Friday October 18, 2019 @05:07AM
(#21289796)
In-kernal block caching of files is pretty crude - since the kernel has no knowledge of your data structures or which bits of your files you're going to be reading next, you end up having hundreds of page faults requiring single-sector disk reads for most workloads.
That’s often the case if your file format evolves without taking a mmap use case into account. But for a format that’s designed with mmapping in mind, it is often ridiculously faster and more effective.
But most languages and environments don’t let you do that easily.
Especially if you can't or don't want to use memory mapping because it's hard to do well.
But more importantly, I think that the server can monitor file changes ahead of time with things like inotify, saving the time of stat()-ing files when the user wants to perform a build.