News for hackers. Stuff that matters.
Re:Monoliths Are the Future by te_chris on Thursday January 30, 2020 @05:48PM (#22196752)
The data lake is also a real, live GDPR PII time bomb if you worked out how to get the data in but not take it out
Re:Monoliths Are the Future by lolc on Thursday January 30, 2020 @07:25PM (#22197616)
But if I haven't spent the effort to extract it, do I really own it? Let me argue that I don't have it because all my implemented queries turn up none of your data. You wouldn't tax me on gold that hasn't yet been extracted, would you? (End of joke.)
Re:Monoliths Are the Future by ska on Thursday January 30, 2020 @07:48PM (#22197785)

  But if I haven't spent the effort to extract it, do I really own it? 
If you collected it, you are responsible for it.
Re:Monoliths Are the Future by yesbabyyes on Thursday January 30, 2020 @08:05PM (#22197929)
What I think you'd typically do is put different data under different keys/paths, so that red is personally identifiable data, yellow contains pointers to such data, and green is just regular data. You could have a structure like s3://my-data-lake/{red|yellow|green}/{raw|intermediate}/year={year}/month={month}/day={day}/source={system}/dataset={table}

Then you just don't keep red data for longer than 30 days.