Ask HN: How to build data pipelines to continuously ingest customer files?
I'm working on onboarding a customer who wants to deliver their data as a weekly dump of CSVs. We’re supposed to ingest that data and get it into our system to provide analytics for their team.I was initially thinking of just building a one-off ETL script, but I was warned the files may randomly break spec (new/renamed fields, etc) due to errors in the process that generates them. Is there a...