News for hackers. Stuff that matters.
Re:But how, exactly, do databases use mmap? by cperciva on Saturday January 23, 2021 @12:39PM (#25884354)
There's an even better reason for databases to not write to memory mapped pages: Pages get synched out to disk at the kernel's leisure. This can be ok for a cache but it's definitely not what you want for a database!
Re:But how, exactly, do databases use mmap? by eqvinox on Saturday January 23, 2021 @01:14PM (#25884700)
That's what msync() is for.
Re:But how, exactly, do databases use mmap? by cperciva on Saturday January 23, 2021 @03:12PM (#25885613)
If you're tracking what needs to be flushed to disk when, you might as well just be making explicit pwrite syscalls.
Re:But how, exactly, do databases use mmap? by monocasa on Saturday January 23, 2021 @01:21PM (#25884769)
Right, but it can sync arbitrary ranges sooner, which is also awful for consistency.
Re:But how, exactly, do databases use mmap? by reader_mode on Saturday January 23, 2021 @02:08PM (#25885145)
Shouldn't your write strategy be resilient to that kind of stuff (eg. shutdown during a partial update) ?
Re:But how, exactly, do databases use mmap? by gmueckl on Saturday January 23, 2021 @02:36PM (#25885344)
Don't you need exact guarantees on write ordering to achieve that?
Re:But how, exactly, do databases use mmap? by jorangreef on Saturday January 23, 2021 @02:53PM (#25885480)
Yes, for almost all databases, although there was a cool paper from the University of Wisconsin Madison a few years ago that showed how to design something that could work without write barriers, and under the assumption that disks don't always fsync correctly:

"the No-Order File System (NoFS), a simple, lightweight file system that employs a novel technique called backpointer based consistency to provide crash consistency without ordering writes as they go to disk"

http://pages.cs.wisc.edu/~vijayc/nofs.htm

Re:But how, exactly, do databases use mmap? by vlovich123 on Saturday January 23, 2021 @03:45PM (#25885891)
Does that generalize to databases? My understanding is that file systems are a restricted case of databases that don’t necessarily support all operations (eg transactions are smaller, can’t do arbitrary queries within a transaction, etc etc).
Re:But how, exactly, do databases use mmap? by jorangreef on Sunday January 24, 2021 @12:09AM (#25889476)
I think it would, because if your underlying file system can provide crash consistency then that can be leveraged as a primitive by the database.
Re:But how, exactly, do databases use mmap? by bonzini on Saturday January 23, 2021 @02:48PM (#25885433)
You can do write/sync/write/sync in order to achieve that. It would be nicer to have FUA support in system calls (or you can open the same file to two descriptors, one with O_SYNC and one without).
Re:But how, exactly, do databases use mmap? by dooglius on Saturday January 23, 2021 @05:47PM (#25886857)
I think you mean mlock