News for hackers. Stuff that matters.
Re:ETL Pipelines with Airflow: The Good, the Bad and the Ugly by wara23arish on Friday October 08, 2021 @12:28PM (#28801111)
Are you talking about Apache Beam? I happened to land a job where I had learned how to create dataflow pipelines using Apache Beam (java) on gcp.

Im a little worried that I might be investing my time on a skill/tool that isn’t that much in demand. (Instead of common backend/frontend development)

Re:ETL Pipelines with Airflow: The Good, the Bad and the Ugly by joeswartz on Friday October 08, 2021 @06:08PM (#28804702)
Apache Beam is a quite nice and flexible tool. If you prefer Scala, then you can try Scio: https://github.com/spotify/scio.