Re:Fastest table sort in the West – Redesigning DuckDB's sort
by nextaccountic on Friday August 27, 2021 @07:10PM
(#28333993)
> I'm not sure if I understand, but we did not use a memory map (mmap), but rather blocks of memory that are explicitly (un)loaded by the buffer manager.
But then, do you have your own cache competing with the OS cache? Or you use O_DIRECT to disable the filesystem cache (at least on Linux)?
Re:Fastest table sort in the West – Redesigning DuckDB's sort
by tomnipotent on Friday August 27, 2021 @09:30PM
(#28334783)
Most databases rely on OS filesystem cache, which is very compatible with the way buffer caches manage data with fixed-size pages/blocks.
Re:Fastest table sort in the West – Redesigning DuckDB's sort
by jandrewrogers on Friday August 27, 2021 @11:31PM
(#28335402)
Some databases rely on the OS file system cache, this is true. However, it is not controversial that doing so reduces performance by integer factors. Bypassing the OS file system cache is idiomatic in database kernel designs where absolute performance is a key objective.
Re:Fastest table sort in the West – Redesigning DuckDB's sort
by tomnipotent on Sunday August 29, 2021 @03:06AM
(#28344823)
Using O_DIRECT and other features still makes you reliant on the OS cache, even if all you're doing is keeping it empty and out of your business. I've only run into a few that manually manage block devices directly.
The M1 + SSD performs really well here. We tried to an external sort experiment on the x86 machine, but the SSD is old and only has a write speed of 150MB/s (compared to the MacBook's 3GB/s) and it was incredibly slow. So you definitely need a fast SSD for this.
The columns we chose to sort by are rather arbitrary, but we shuffled the table before running the experiments to make sure there is no ordering left from the generation in there.
I like the merge sort trick you described!