News for hackers. Stuff that matters.

A comment on Build your own “data lake” for reporting purposes. Parent comment

Re:Build your own “data lake” for reporting purposes by ekianjo on Monday March 15, 2021 @07:13AM (#26464523)
You do not even need to use AWS. use Minio as S3 compatible system and Nifi to convert files to parquet once they land in Minio... No dependency on AWS.
[ Parent | on HN ]
Re:Build your own “data lake” for reporting purposes by antman on Monday March 15, 2021 @11:21AM (#26467052)
Minio and nifi, require lots of resources for themselves. Better off using pure python and if one wants something lighweight and visually pleasing Mara [0] or Dagster with Dagit [1] will do the job

[0] https://github.com/mara/mara-pipelines

[1] https://docs.dagster.io/tutorial/execute

[ Parent | on HN ]
Re:Build your own “data lake” for reporting purposes by ekianjo on Monday March 15, 2021 @05:53PM (#26469683)
Yes, Minio and Nifi are not lightweight but if you look for robustness they have been used in large environments and proven to scale.
[ Parent | on HN ]