News for hackers. Stuff that matters.
Re:The Bw-Tree: A B-tree for New Hardware by hyc_symas on Wednesday April 10, 2013 @11:18AM (#5525646)
Better to make everything as simple and as easy as possible. App-managed chunking is error-prone, and it forces redundancy of implementations. Likewise, "you have an extra in memory copy" can make or break a system. OpenLDAP slapd using LMDB uses 1/4 the RAM of slapd using BerkeleyDB for a given number of entries. That means you can serve 4x as many entries using LMDB on a given hw config at full RAM speed, vs BDB. LMDB efficiency has a direct impact on hardware cost and electricity cost, as well as sw development cost.
Re:The Bw-Tree: A B-tree for New Hardware by jules on Wednesday April 10, 2013 @11:47AM (#5525834)
Exactly! So do real world fair benchmarks that show 4x improvement, not unfair benchmarks that literally show 3000x improvement over BDB.
Re:The Bw-Tree: A B-tree for New Hardware by hyc_symas on Wednesday April 10, 2013 @05:12PM (#5528215)
We are doing all of the above. Microbenchmarks that show only the performance of the DB engine still serve a purpose. And the difference between the microbenches and the full benches shows you how much overhead is eaten in the rest of the system. E.g., when testing read performance of SQLite3 adapted to LMDB, we measured only a 1-2% improvement, because the majority of SQLite3 read execution time is spent in SQL processing, not in its Btree code.
Re:The Bw-Tree: A B-tree for New Hardware by hyc_symas on Wednesday April 10, 2013 @05:37PM (#5528371)
Here's some more real world testing, done by the folks at Zimbra:

http://wiki.zimbra.com/wiki/OpenLDAP_MDB_vs_HDB_performance

Clearly more than 4x write speed improvement.

Re:The Bw-Tree: A B-tree for New Hardware by jules on Thursday April 11, 2013 @06:02PM (#5535062)
Nice! Do you know why that benchmark show almost no read improvement but massive write improvement? The slides here (http://symas.com/mdb/20120322-UKUUG-MDB.pdf) suggest the opposite? Or am I reading them incorrectly?
Re:The Bw-Tree: A B-tree for New Hardware by hyc_symas on Friday April 12, 2013 @04:36AM (#5537281)
As I understand it, that test used something like 32 server threads on a machine with only 4 cores. The UKUUG prezo used 16 server threads on a machine with 16 cores, which allowed all of the reads to run concurrently. The smaller number of CPU cores in the Zimbra test naturally reduces the concurrency advantage of LMDB.