News for hackers. Stuff that matters.
Re:1.1B Taxi Rides Using OmniSciDB and a MacBook Pro by nikita on Wednesday July 29, 2020 @10:13PM (#23993695)
This sounds about right. The query is using count and not sum which i believe can be a bit faster.

Yes. Branching will absolutely hurt. Good old x100 paper teaches how to avoid branching: http://cidrdb.org/cidr2005/papers/P19.pdf.

And of course there is no branching in MemSQL for this use case. And also no hashing b/c number of groups is small and you can use an array and not a hashtable.

Finally if you compress data rather than do the sum on an uncompressed array you will have a lot more compact data representation which would allow you not hit the memory bandwidth ceiling this quickly (4 threads)

Re:1.1B Taxi Rides Using OmniSciDB and a MacBook Pro by tmostak on Thursday July 30, 2020 @01:01AM (#23994845)
Hi @nikita, good to reconnect.

When you say an array and not a hash table, do you just mean a simple perfect hash table indexed by the offset of the dictionary id? We use this fairly extensively for inputs of bounded domain (i.e. dictionary-encoded strings, moderately-sized integer ranges, even binned values, numeric or timestamp), but call it a perfect hashing. Assume we're talking about the same thing but wanted to clarify.

Re:1.1B Taxi Rides Using OmniSciDB and a MacBook Pro by nikita on Thursday July 30, 2020 @01:20AM (#23994944)
Yes, that’s it.

I’m still of an opinion that it’s important to demonstrate performance on more complex queries with joins, subqueries, subselects, and clustered data movements. The count(*), group by query is a very very simple case.