Re:Lightning Memory-Mapped Database Manager (LMDB) 1.0
by markasoftware on Thursday July 02, 2026 @08:52PM
(#48769335)
FWIW I had this issue even with the MDB_NOSYNC flag so it shouldnt be force flushing to disk unless I'm out of ram or whatever
Re:Lightning Memory-Mapped Database Manager (LMDB) 1.0
by hyc_symas on Friday July 03, 2026 @11:18AM
(#48776081)
LMDB 1.0 no longer uses a P_DIRTY flag, it no longer has to explicitly mark pages as clean.
Re:Lightning Memory-Mapped Database Manager (LMDB) 1.0
by keynha on Sunday July 05, 2026 @02:44AM
(#48791822)
Dropping the explicit P_DIRTY flag in 1.0 is a neat change. What tracks which pages still need to be written back at commit now that the flag is gone?
Re:Lightning Memory-Mapped Database Manager (LMDB) 1.0
by hyc_symas on Sunday July 05, 2026 @10:32AM
(#48794579)
The txnID was added to the page header to enable support for incremental backup. As a consequence, it's sufficient to compare a page's txnID to the current txnID to know if it's dirty or not, and spilled pages don't need a cleanup pass to clear their dirty bit on commit so commits of large txns are faster now.
Re:Lightning Memory-Mapped Database Manager (LMDB) 1.0
by hyc_symas on Saturday July 04, 2026 @05:48AM
(#48784166)
By the way, you're wrong on both points - the cache of page mallocs is not infinite, and it does spill dirty pages to disk when necessary. And the latter is what bounds the number of malloc'd pages.
Idk, if it was your issue, but for long running write transactions it doesn't spill to disk. So you have all the changes being written to disk at the end of the transaction. One would think enabling write mapping fixes this, but it needs to mark all the pages as clean before commit, so same effect there. I fixed this for 0.9 here https://github.com/uroni/hs5/tree/main/external/lmdb . Will have to investigate if it is improved with 1.0, or if I need to redo the changes.
Edit: Just noticed that the issue is about free list in the file. Never had a problem with that, but I also had to replace that MIDL structure with something more scalable for the spilling.