Re:ArcticDB: Why a Hedge Fund Built Its Own Database
by joewood1972 on Sunday August 25, 2024 @02:07PM
(#41349628)
For example, Apache Iceberg is exactly this. Complete with bitemporal query support.
Re:ArcticDB: Why a Hedge Fund Built Its Own Database
by dan-robertson on Sunday August 25, 2024 @04:13PM
(#41350932)
I feel like ‘exactly’ is doing a lot of work in your comment and I am interested in the reasons that that word may not be quite the right word to describe these situations.
Re:ArcticDB: Why a Hedge Fund Built Its Own Database
by jjmunro on Monday August 26, 2024 @05:18AM
(#41355352)
We do have a specialist time-series index, optimised for things like tick-data. It compresses fairly well but we generally optimise for read-time. Not all over the place random-access, but slicing out date-ranges. There are two layers of index, a high level index of the data-objects, and the index in each object in S3.
A built-in as-of join is something we want to build.
A slightly more technical question is what your time series indexes are? Is it about optimising storage, or doing fast random-access lookups, or more for better as-of joins?