News for hackers. Stuff that matters.
Re:How to Analyze Billions of Records per Second on a Single... by gricardo99 on Monday July 09, 2018 @03:49PM (#17492903)
Yup.

>The pickup_ntaname column is stored as varchars by ClickHouse and kdb+, and as dictionary encoded single byte values by LocustDB.

But it would be trivial to convert to enum/sym type in kdb+. It's silly to query and group by strings.

Re:How to Analyze Billions of Records per Second on a Single... by inteleng on Monday July 09, 2018 @05:44PM (#17493689)
This is one of the frustrating parts of database software. Where can information about the ways to optimize these parameters be found (outside of random posts scattered around StackExchange)?
Re:How to Analyze Billions of Records per Second on a Single... by gricardo99 on Tuesday July 10, 2018 @01:31PM (#17500311)
so true. Either you have lots of experience working with specific databases to setup/optimize queries, and you know what works best through personal blood, sweat and tears, and/or you have intimate knowledge of the inner workings of the database architecture/implementation and know the theoretical best approach to structure your schema/queries.

But even then, hardware/networking performance and tuning can throw a wrench in the most seasoned/knowledgeable approaches. Users can further bring otherwise solid setups to a grinding halt with unanticipated use-cases.

The only hope when you hit these inevitable road-blocks is that you're working for someone that appreciates the difficulty of the problem.