News for hackers. Stuff that matters.
Re:Comparison of Data Lake Table Formats (Iceberg, Hudi and ... by eatonphil on Monday June 13, 2022 @06:48PM (#31732719)
You can do this for small datasets and I built an app around exactly this use case [0].

As your data gets bigger (just over time even if individual days don't emit more data) you'll end up building out partitioning schemes yourself or you'll move to a system that does that for you like bigquery/snowflake/etc.

[0] https://github.com/multiprocessio/datastation

There are no comments on this comment yet.