Re:The final step for huge-page swapping
by sterlind on Wednesday July 04, 2018 @01:17AM
(#17455390)
the feature you're looking for is called opening a memory mapped file. I've seen tremendous performance improvements, but it's a really underused feature for some reason. Memory map Performance is incredible.
[ Parent
| on HN ]
Re:The final step for huge-page swapping
by bluejekyll on Wednesday July 04, 2018 @10:19AM
(#17457394)
> it's a really underused feature for some reason
My suspicion is that itβs mainly because memory mapping files, while safe for reading, have a whole bunch of edge cases where the file can accidentally be corrupted.
Does Windows have any analog to madvise()? It seems like that's probably necessary to making things work well when you're mapping huge data sets, unless you're sure they reside on really fast storage.
Windows 8 and later has PrefetchVirtualMemory[1]. Before that your best bet is to get a little hacky and try reading the file asynchronously to get it in the cache (Windows caches files pretty aggressively, so this more or less works).
[1] https://msdn.microsoft.com/en-us/library/windows/desktop/hh7...
Re:The final step for huge-page swapping
by senozhatsky on Wednesday July 04, 2018 @04:08AM
(#17455901)
Seems that VirtualAlloc() [1] is a little bit closer to madvise(), but still is pretty far. madvise() is quite
powerful. For instance, take a look at MADV_FREE.
[1] https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
-ss
I figured the use-case here was MADV_WILLNEED, which is what PrefetchVirtualMemory is for. VirtualAlloc flags would be for things like reserving virtual pages without committing them.
There's no single API on Windows that does the myriad of things that madvise does.