News for hackers. Stuff that matters.

A comment on Build your own “data lake” for reporting purposes. Parent comment

Re:Build your own “data lake” for reporting purposes by unixhero on Monday March 15, 2021 @05:47PM (#26469602)
What does Parquet do for your problems? I am trying to learn for when I might need it?
[ Parent | on HN ]
Re:Build your own “data lake” for reporting purposes by gizmodo59 on Tuesday March 16, 2021 @09:06AM (#26476183)
There are several advantages compared to CSV, JSON or other non-columnar formats as far as analytics is concerned which are typically not transactional.

https://blog.openbridge.com/how-to-be-a-hero-with-powerful-p... https://stackoverflow.com/a/36831549

And with projects like Apache Iceberg you can also have ACID transactions which will make it easy to update or delete just using SQL. This really opens up the separation of compute and data and you can use any engine you want (spark, drill, hive, impala, athena, dremio, redshift spectrum etc) on top of your files.

[ Parent | on HN ]