News for hackers. Stuff that matters.
Re:Daft: A High-Performance Distributed Dataframe Library fo... by rgrieselhuber on Wednesday June 07, 2023 @08:40AM (#36226048)
2. I hate the name data lake but I’ve always used them as part of a pipeline. It’s useful to keep the raw data around when you need to recover / replay the pipeline.
Re:Daft: A High-Performance Distributed Dataframe Library fo... by short_sells_poo on Wednesday June 07, 2023 @08:57AM (#36226217)
I think most data lakes really turn into a data swamp as people/orgs are not diligent enough to keep them in a good shape. In absence of costs, people never delete anything voluntarily and the garbage will grow monotonically until it takes up all space.
Re:Daft: A High-Performance Distributed Dataframe Library fo... by musingsole on Wednesday June 07, 2023 @09:12AM (#36226370)
That applies to databases as much as data lakes.

I don't know why it's so hard to get across that data expires in the same way that language changes.