Building columnar compression in a row-oriented database

Re:Building columnar compression in a row-oriented database by RobAtticus on Thursday October 31, 2019 @04:52PM (#21413609)

For choosing the column, you'll usually want to think about what your queries will be using. It sounds like `recorded_at` is probably more likely to be useful since that's when the data "occurred," but again it depends on your expected query load.

Out of order data should be handled fine by TimescaleDB -- if you do have data that is far in the future or in the past, you may get stray chunks to hold those, but it's not going to create all the intermediate chunks or anything that might be undesirable. You can later correct those fields by deleting and reinserting the record with a corrected timestamp.

[ Parent | on HN ]

Re:Building columnar compression in a row-oriented database by jnordwick on Thursday October 31, 2019 @05:02PM (#21413682)

Sounds like he's describing a bitemporal database (although not in the canonical form usually associated with them).

I've looked into timescaledb for this, and it doesn't support them.

[ Parent | on HN ]

Re:Building columnar compression in a row-oriented database by refset on Thursday October 31, 2019 @05:42PM (#21414007)

I agree. Bitemporal databases can natively handle late-arriving data in these kinds of upstream timestamp integration scenarios.

However, the intersection of bitemporal indexes and columnar time-series queries seems important and yet I haven't seen anything that looks like it might offer both, possibly asides from kdb+ and SAP HANA.

Disclosure: I work on https://github.com/juxt/crux (which is optimised for bitemporal graph joins and doesn't currently employ columnar indexes)

[ Parent | on HN ]