Snel: SQL Native Execution for LLVM

Re:Snel: SQL Native Execution for LLVM by alecco on Monday February 24, 2020 @01:40PM (#22406958)

Actually for the case it was on SSD. Most of the access is to compressed columnar data. The main table had 500+ columns of ~140m rows but with very low entropy (e.g. gender, city/district). Each column had its own file. And queries were relatively simple. At 300MB/s (at the time) it was pretty fast. Data was mmap since as in most real life scenarios, the access to the columns had a power law distribution. Most queries could resolve with RAM data (~24GB/s at the time). Data size was over 10x the size of RAM (~128GB). IO is fast if you do mostly sequential access. Note server NVMe today is ~8x faster so the principle still applies.

The main thing here is the simplicity of the architecture and combining HyPer, careful mmap usage, and having SQLite engine do all the other non-critical work. So this way this system could be implemented on time by 1 main developer (rockstar dev, Marcelo) and a tiny little help from me.

(Actual author of the idea/architecture here, not mentioned on the paper, but that's life, haha)

[ Parent | on HN ]

Re:Snel: SQL Native Execution for LLVM by ww520 on Monday February 24, 2020 @07:18PM (#22409787)

That's pretty sweet information. For columnar db, LLVM definitely helps as columnar wants to cram as much small type data as possible in memory and runs the CPU over them. I would imagine the condition evaluation in the where-clause would be benefited the most since it's mostly an evaluation of dynamic expression.

[ Parent | on HN ]

Re:Snel: SQL Native Execution for LLVM by blattimwind on Monday February 24, 2020 @02:06PM (#22407254)

> Note server NVMe today is ~8x faster so the principle still applies.

But memory isn't. With NVMe the disk-memory-gap has shrunk considerably over previous SSDs and HDD RAIDs.

[ Parent | on HN ]