News for hackers. Stuff that matters.

A comment on Ask HN: Is it a good idea to use AWS Lambda for ETL Pipelines?.

Re:Ask HN: Is it a good idea to use AWS Lambda for ETL Pipel... by ignorantguy on Monday January 17, 2022 @01:25PM (#29969816)
We use lambda in our production environment for ETL. It works really well. In our case when we were using it, we modeled the code to account for 15 minute window. Hope that helps.
[ Parent | on HN ]
Re:Ask HN: Is it a good idea to use AWS Lambda for ETL Pipel... by 91dylwyn on Monday January 17, 2022 @01:34PM (#29970001)
Are you finding any pain points? How did you manage the 15 minute window in case of large amounts of data running over time?

How do you manage the DAGs on it? For example creating dependencies between tasks and monitoring and rerunning tasks?

[ Parent | on HN ]