Re:How to Analyze Billions of Records per Second on a Single...
by inteleng on Monday July 09, 2018 @05:44PM
(#17493689)
This is one of the frustrating parts of database software. Where can information about the ways to optimize these parameters be found (outside of random posts scattered around StackExchange)?
Re:How to Analyze Billions of Records per Second on a Single...
by gricardo99 on Tuesday July 10, 2018 @01:31PM
(#17500311)
so true. Either you have lots of experience working with specific databases to setup/optimize queries, and you know what works best through personal blood, sweat and tears, and/or you have intimate knowledge of the inner workings of the database architecture/implementation and know the theoretical best approach to structure your schema/queries.
But even then, hardware/networking performance and tuning can throw a wrench in the most seasoned/knowledgeable approaches. Users can further bring otherwise solid setups to a grinding halt with unanticipated use-cases.
The only hope when you hit these inevitable road-blocks is that you're working for someone that appreciates the difficulty of the problem.
>The pickup_ntaname column is stored as varchars by ClickHouse and kdb+, and as dictionary encoded single byte values by LocustDB.
But it would be trivial to convert to enum/sym type in kdb+. It's silly to query and group by strings.