News for hackers. Stuff that matters.

A comment on Accidentally exponential behavior in Spark. Parent comment

Re:Accidentally exponential behavior in Spark by mjburgess on Tuesday June 22, 2021 @05:15AM (#27589533)
Is there anything you can say about Spark for Data Engineering (/ETL) ?

The most common reason for spark use today is ETL+DataLakes (ie., cloud object stores and ETL in/out).

It seems actual analysis is happening in fast databases that receive data from the object stores.

can anyone here comment on this paradigm?

[ Parent | on HN ]
Re:Accidentally exponential behavior in Spark by xentripetal on Tuesday June 22, 2021 @04:41PM (#27596904)
I don't have much insight into spark but I've been using Dataflow/beam for ETL. Been a pretty good experience. follows the style of spinning up compute to process as needed then shutdown.
[ Parent | on HN ]