Large-Scale Transactional Data Lake at Uber Using Apache Hudi

Re:Large-Scale Transactional Data Lake at Uber Using Apache ... by teraku on Wednesday June 10, 2020 @07:04AM (#23475780)

Spark being an example of a tool that (also) exists inside the Hadoop ecosystem and is still gaining acceptance due to popularity, but at the same time losing traction as better option are arising.

[ Parent | on HN ]

Re:Large-Scale Transactional Data Lake at Uber Using Apache ... by rory_isAdonk on Wednesday June 10, 2020 @07:36AM (#23475942)

What better options? Can you drop some links? Thanks :)

[ Parent | on HN ]

Re:Large-Scale Transactional Data Lake at Uber Using Apache ... by teraku on Wednesday June 10, 2020 @11:56AM (#23478543)

So the main issue with Spark is that the streaming is not that great. In general a Spark cluster persé is not bad, but I see more and more people using Apache Beam and then just use a runner that either they already have or that fits them best. https://beam.apache.org/

[ Parent | on HN ]

Re:Large-Scale Transactional Data Lake at Uber Using Apache ... by eunos on Wednesday June 10, 2020 @12:20PM (#23478852)

Apache Flink

[ Parent | on HN ]

Re:Large-Scale Transactional Data Lake at Uber Using Apache ... by StreamBright on Wednesday June 10, 2020 @09:48AM (#23476974)

Depending on cloud vs on-prem.

[ Parent | on HN ]

Re:Large-Scale Transactional Data Lake at Uber Using Apache ... by teraku on Wednesday June 10, 2020 @11:58AM (#23478568)

Not really, just use Apache Beam and set the runner to whatever you have / prefer

[ Parent | on HN ]

Re:Large-Scale Transactional Data Lake at Uber Using Apache ... by StreamBright on Wednesday June 10, 2020 @04:23PM (#23481627)

No thanks. I do not see any value of the n+1 unnecessary abstraction over things that I am already familiar with. An average customer does not want to use these things either.

[ Parent | on HN ]