Re:Comparison of Data Lake Table Formats (Iceberg, Hudi and ...
by eatonphil on Monday June 13, 2022 @06:48PM
(#31732719)
You can do this for small datasets and I built an app around exactly this use case [0].
As your data gets bigger (just over time even if individual days don't emit more data) you'll end up building out partitioning schemes yourself or you'll move to a system that does that for you like bigquery/snowflake/etc.
As your data gets bigger (just over time even if individual days don't emit more data) you'll end up building out partitioning schemes yourself or you'll move to a system that does that for you like bigquery/snowflake/etc.
[0] https://github.com/multiprocessio/datastation