News for hackers. Stuff that matters.

A comment on We reduced the AWS costs of our streaming data pipeline. Parent comment

Re:We reduced the AWS costs of our streaming data pipeline by runT1ME on Thursday May 28, 2020 @12:39AM (#23333451)
If you're already paying the cost(both engineering time and compute wise) for EMR, I can't imagine it takes more effort to create a new Flink job than a new Glue job?

The advantage of Glue or the corresponding serverless GCP ETL option (dataflow) is that it's serverless elastic, but it sounds like their workload wasn't applicable.

[ Parent | on HN ]
Re:We reduced the AWS costs of our streaming data pipeline by brodouevencode on Thursday May 28, 2020 @09:36AM (#23337044)
Unfortunately that's not nearly how AWS works. AWS breaks down everything and charges you for it separately. Flink and Glue are entirely different animals.
[ Parent | on HN ]
Re:We reduced the AWS costs of our streaming data pipeline by runT1ME on Thursday May 28, 2020 @11:57PM (#23346130)
Can you explain more? From what I understand, if you're already running a Flink cluster on AWS and you have capacity for another job, you aren't charged more, no?

I haven't used Glue, but it seems like it's able to do stream processing on Kinesis and dumping to S3 or whatnot, so it seems like there's overlap with using EMR running Flink?

[ Parent | on HN ]