Build your own “data lake” for reporting purposes

Re:Build your own “data lake” for reporting purposes by gizmodo59 on Sunday March 14, 2021 @09:44PM (#26461691)

For my home projects I generate parquet (columnar and very well suited for DW like queries) files with pyarrow and use Dremio (Direct SQL on data lake): https://github.com/dremio/dremio-oss (https://www.dremio.com/on-prem/) to query them (minio or just local disk or s3) and use Apache Superset for quick charts or dashboards.

[ Parent | on HN ]

Re:Build your own “data lake” for reporting purposes by unixhero on Monday March 15, 2021 @05:47PM (#26469602)

What does Parquet do for your problems? I am trying to learn for when I might need it?

[ Parent | on HN ]

Re:Build your own “data lake” for reporting purposes by gizmodo59 on Tuesday March 16, 2021 @09:06AM (#26476183)

There are several advantages compared to CSV, JSON or other non-columnar formats as far as analytics is concerned which are typically not transactional.

https://blog.openbridge.com/how-to-be-a-hero-with-powerful-p... https://stackoverflow.com/a/36831549

And with projects like Apache Iceberg you can also have ACID transactions which will make it easy to update or delete just using SQL. This really opens up the separation of compute and data and you can use any engine you want (spark, drill, hive, impala, athena, dremio, redshift spectrum etc) on top of your files.

[ Parent | on HN ]