Lessons learned from running Apache Airflow at scale

Re:Lessons learned from running Apache Airflow at scale by taude on Monday May 23, 2022 @01:07PM (#31481748)

Tangentially to this thread....what sites, sources, etc. are people who work on modern data pipelines (engineering and analysts) going to follow the latest news, products, techniques, etc. It's been hard to keep up without having Meetups and such the last couple years. I'm finding a lot of people's comments here pretty interesting, and showing me things I haven't heard of. Thanks.

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by blakeburch on Monday May 23, 2022 @05:28PM (#31484971)

I've had really great success from engaging with the Locally Optimistic Slack community.

Also, Cristophe Blefari has an excellent data newsletter. https://www.blef.fr/

And Modern Data Stack has a newsletter, tool information, Q&A www.moderndatastack.xyz

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by mcnnowak on Monday May 23, 2022 @01:16PM (#31481885)

I'm also interested in this topic, but can't find anything other than "Top 10 things you should STOP doing as a data engineer" etc. content-mill, clickbait on Medium and other sites.

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by taude on Monday May 23, 2022 @01:32PM (#31482122)

Yes, this. I'd like to get less of the "Marketing sales stuff", and more in the trenches with the actual engineering teams.

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by dtjohnnyb on Monday May 23, 2022 @01:20PM (#31481937)

Slack groups have filled in the meetup space in my life, mlops.community and locally optimistic are two of the best for what it sounds like you're looking for

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by kderbe on Monday May 23, 2022 @01:48PM (#31482394)

I follow the Analytics Engineering Roundup weekly email. It's published by dbt Labs but isn't overtly promotional.

https://roundup.getdbt.com/

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by taude on Monday May 23, 2022 @02:02PM (#31482643)

Thanks. We're starting to use DBT, too. I know the forums over at DBT are pretty good, too.

[ Parent | on HN ]

Re:Lessons learned from running Apache Airflow at scale by jonpon on Monday May 23, 2022 @03:40PM (#31483963)

Data Twitter and Linkedin are great, there are a lot of people putting out some really good content. There are also a lot of substacks you can sign up for. Data Engineering Weekly is my fave

[ Parent | on HN ]