News for hackers. Stuff that matters.
Re:1.1B Taxi Rides Using OmniSciDB and a MacBook Pro by bluestreak on Wednesday July 29, 2020 @08:39PM (#23993137)
Thanks for the explanation Nikita. Any branching, hashing etc will increase execution time. The execution example is on laptop, which has 2 memory channels. 0.13s is absolute max that this laptop is able to muster as far as memory throughput goes. Usually 4 threads is enough to saturate memory.

I have written this piece of code: https://github.com/questdb/questdb/blob/master/core/src/main...

This sums 64bit values and using AVX2 it will sum 1Bn in 0.26s. Incrementing conditionally will not be as fast and will throw vectorization out of the window too.

Re:1.1B Taxi Rides Using OmniSciDB and a MacBook Pro by nikita on Wednesday July 29, 2020 @10:13PM (#23993695)
This sounds about right. The query is using count and not sum which i believe can be a bit faster.

Yes. Branching will absolutely hurt. Good old x100 paper teaches how to avoid branching: http://cidrdb.org/cidr2005/papers/P19.pdf.

And of course there is no branching in MemSQL for this use case. And also no hashing b/c number of groups is small and you can use an array and not a hashtable.

Finally if you compress data rather than do the sum on an uncompressed array you will have a lot more compact data representation which would allow you not hit the memory bandwidth ceiling this quickly (4 threads)

Re:1.1B Taxi Rides Using OmniSciDB and a MacBook Pro by tmostak on Thursday July 30, 2020 @01:01AM (#23994845)
Hi @nikita, good to reconnect.

When you say an array and not a hash table, do you just mean a simple perfect hash table indexed by the offset of the dictionary id? We use this fairly extensively for inputs of bounded domain (i.e. dictionary-encoded strings, moderately-sized integer ranges, even binned values, numeric or timestamp), but call it a perfect hashing. Assume we're talking about the same thing but wanted to clarify.

Re:1.1B Taxi Rides Using OmniSciDB and a MacBook Pro by nikita on Thursday July 30, 2020 @01:20AM (#23994944)
Yes, that’s it.

I’m still of an opinion that it’s important to demonstrate performance on more complex queries with joins, subqueries, subselects, and clustered data movements. The count(*), group by query is a very very simple case.