Using SIMD to aggregate billions of values per second

Re:Using SIMD to aggregate billions of values per second by smabie on Tuesday April 07, 2020 @09:59PM (#22809389)

The reason why you cannot ever compete with kdb+/q is because the database and language run in one address space. Your benchmark gets around this problem by using the built in sum() function, but kdb+/q can just execute arbitrary code and never suffer a performance penalty. Unless you plan on integrated a high performance programming language into your DB, it simply will not be possible to ever meaningfully compete on the effective total time of complex queries.

I, of course, am not disparaging your work, the performance numbers are very impressive!

[ Parent | on HN ]

Re:Using SIMD to aggregate billions of values per second by patrick73_uk on Wednesday April 08, 2020 @05:26AM (#22811369)

I'm a QuestDB dev, data on a QuestDb is also stored in a single address space and SQL queries are compiled into objects that run in that address space. However, it is just SQL not a bespoke language. A future possibility would be to allow queries in java or scala.

[ Parent | on HN ]

Re:Using SIMD to aggregate billions of values per second by smabie on Wednesday April 08, 2020 @02:52PM (#22815799)

Implementing the ability for QuestDB to dynamically load jars would be really cool. And if you exposed an interface to directly communicate with the Db, you could get rid of the SQL parsing overhead as well. This would also allow QuestDB to function as an app engine of sorts, just like kdb+/q. I see real value in that for latency sensitive financial applications.

[ Parent | on HN ]