When your data doesn’t fit in memory: the basic techniques

Re:When your data doesn’t fit in memory: the basic techniques by Animats on Tuesday November 12, 2019 @02:56AM (#21512334)

Um, yes. From the article:

"The simplest and most common way to implement indexing is by naming files in a directory:"

    mydata/
        2019-Jan.csv
        2019-Feb.csv
        2019-Mar.csv
        2019-Apr.csv

Um.

[ Parent | on HN ]

Re:When your data doesn’t fit in memory: the basic techniques by kyllo on Tuesday November 12, 2019 @11:41AM (#21515477)

This is time partitioning, not indexing, but it's commonly done in data lakes for raw telemetry/log data.

    mydata/
        2019/
            11/
               12/
                   2019111200.csv
                   2019111201.csv

Where each one of those CSV files could be 10-100 GB in size.

Usually you want to process it into a columnar format like Parquet from there, though

[ Parent | on HN ]