Re:Monoliths Are the Future
by ForHackernews on Thursday January 30, 2020 @03:08PM
(#22194967)
What/where do you run this mythical one-analyst pipeline, though? Is that in cloud services too? Airflow? Kubeflow? Apache Beam? It sounds like you're just pushing the problem around.
[ Parent
| on HN ]
AWS data pipelines and AWS lambda. It's cloud services the whole way down.
I saved a company 20k a month by creating a job server in AWS. Lambda isnt cheap when you start using it hard
Lambda is mostly used for it's trigger functionality for data or artifacts that are created at irregular intervals. Eg, an object is uploaded to s3 which triggers a lambda which runs a copy command for that object into redshift. The kind of stuff that's well below the threshold for leaving the free tier.