News for hackers. Stuff that matters.
Re:The final step for huge-page swapping by sterlind on Wednesday July 04, 2018 @01:17AM (#17455390)
the feature you're looking for is called opening a memory mapped file. I've seen tremendous performance improvements, but it's a really underused feature for some reason. Memory map Performance is incredible.
Re:The final step for huge-page swapping by bluejekyll on Wednesday July 04, 2018 @10:19AM (#17457394)
> it's a really underused feature for some reason

My suspicion is that it’s mainly because memory mapping files, while safe for reading, have a whole bunch of edge cases where the file can accidentally be corrupted.

Re:The final step for huge-page swapping by wtallis on Wednesday July 04, 2018 @01:26AM (#17455418)
Does Windows have any analog to madvise()? It seems like that's probably necessary to making things work well when you're mapping huge data sets, unless you're sure they reside on really fast storage.
Re:The final step for huge-page swapping by PeCaN on Wednesday July 04, 2018 @02:24AM (#17455584)
Windows 8 and later has PrefetchVirtualMemory[1]. Before that your best bet is to get a little hacky and try reading the file asynchronously to get it in the cache (Windows caches files pretty aggressively, so this more or less works).

[1] https://msdn.microsoft.com/en-us/library/windows/desktop/hh7...

Re:The final step for huge-page swapping by senozhatsky on Wednesday July 04, 2018 @04:08AM (#17455901)
Seems that VirtualAlloc() [1] is a little bit closer to madvise(), but still is pretty far. madvise() is quite powerful. For instance, take a look at MADV_FREE.

[1] https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...

-ss

Re:The final step for huge-page swapping by PeCaN on Wednesday July 04, 2018 @01:08PM (#17458298)
I figured the use-case here was MADV_WILLNEED, which is what PrefetchVirtualMemory is for. VirtualAlloc flags would be for things like reserving virtual pages without committing them.

There's no single API on Windows that does the myriad of things that madvise does.

Re:The final step for huge-page swapping by ddorian43 on Wednesday July 04, 2018 @01:29AM (#17455427)
Yep, see lmdb read performance. Or many search engines doing mmap.