News for hackers. Stuff that matters.

A comment on Build your own “data lake” for reporting purposes.

Re:Build your own “data lake” for reporting purposes by pupdogg on Sunday March 14, 2021 @09:58PM (#26461781)
This is an overly complex solution that we were able to resolve using a simple VPS running Clickhouse as backend and Grafana for frontend. Our production db is an Aurora MySQL instance and we keep it lean by performing daily dumps of reporting related data into a CSV with gzip compression -> push it to S3 -> convert it to parquet file format using AWS glue -> bring it into ClickHouse. Data size for these specific reports is approx 100k rows daily and is partitioned by MONTH/YEAR. Overall cost: $20/month VPS and approx. $15/month in AWS billing.
[ Parent | on HN ]
Re:Build your own “data lake” for reporting purposes by twotwotwo on Monday March 15, 2021 @07:31PM (#26470747)
Appreciate the info on real-world use here. A data warehouse would be sort of interesting at work but is not urgently needed (because reporting from MySQL works, without the nifty speedups data warehouses can achieve), and we're somewhere between the size you're talking about and the really-big-data use cases I tend to see blogged about more often. Am curious about ClickHouse and a lower-cost deployment might make it worthwhile when it wouldn't be otherwise.
[ Parent | on HN ]
Re:Build your own “data lake” for reporting purposes by ekianjo on Monday March 15, 2021 @07:13AM (#26464523)
You do not even need to use AWS. use Minio as S3 compatible system and Nifi to convert files to parquet once they land in Minio... No dependency on AWS.
[ Parent | on HN ]
Re:Build your own “data lake” for reporting purposes by antman on Monday March 15, 2021 @11:21AM (#26467052)
Minio and nifi, require lots of resources for themselves. Better off using pure python and if one wants something lighweight and visually pleasing Mara [0] or Dagster with Dagit [1] will do the job

[0] https://github.com/mara/mara-pipelines

[1] https://docs.dagster.io/tutorial/execute

[ Parent | on HN ]
Re:Build your own “data lake” for reporting purposes by ekianjo on Monday March 15, 2021 @05:53PM (#26469683)
Yes, Minio and Nifi are not lightweight but if you look for robustness they have been used in large environments and proven to scale.
[ Parent | on HN ]