News for hackers. Stuff that matters.
Re:A love letter to the CSV format by theLiminator on Wednesday March 26, 2025 @05:31PM (#43487583)
> I've since started looking at parquet files – which turned out to not be friendly to append-only operations. I've ended up implementing writing events into ipc files which then periodically get "flushed" into the parquet files. It works and it's efficient – but man is it non-trivial to implement properly!

I think the industry standard for supporting this is something like iceberg or delta, it's not very lightweight, but if you're doing anything non-trivial, it's the next logical move.

There are no comments on this comment yet.