News for hackers. Stuff that matters.

A comment on Lessons learned from running Apache Airflow at scale. Parent comment

Re:Lessons learned from running Apache Airflow at scale by generalpf on Monday May 23, 2022 @06:47PM (#31485811)
A lot of times this is used for data warehousing so product managers and otherwise can query the database of one app joined with another, especially in an environment with microservices. You might join a table containing orders with another table that was from a totally different DB, like payments, to find out which kinds of items are best to offer BNPL or something.

The author also mentions that it’s used for machine learning models which will ultimately feed back into Shopify’s front end, for instance.

[ Parent | on HN ]
Re:Lessons learned from running Apache Airflow at scale by tootie on Monday May 23, 2022 @08:52PM (#31486741)
I know what a data warehouse is for but this whole situation has doesn't even cover the reporting system which itself is a level removed from any actual product decisions and whether those decisions result in incremental revenue to justify this cost. My company is a fraction the size of Shopify but we have a robust data pipeline and reporting on millions of users run by two people and off the shelf software.
[ Parent | on HN ]