nikhilkuria

Rather than doing a one-time copy of the data, we have applications continually writing to our on-premise MySQL. This data has to reactively reach our datastore on AWS. Also, this is not a 1-1 copy. There are several transformations and cross-validations in play.

There was this one time when we had trouble finding some errors on Cloudwatch logs. This one library was helpful, https://github.com/jorgebastida/awslogs

Thanks! For testing on our local machines, we use SAM Local, https://aws.amazon.com/blogs/aws/new-aws-sam-local-beta-buil... . This has very similar capabilities as local stack. Still, there is always a delay from a service being released by AWS to it being available on tools like Serverless or AWS local.

Exactly. The biggest motivation to use Kinesis was the other services we could plug in from 'AWS ecosystem'. Lambdas have Kinesis triggers available natively. We did not have any concern with the latency of Kinesis. Now I'm curious, do you have any literature on the bad performance history?

Your assumption is correct. One of our goals was to transform and restructure the data we have on our on-site premise MySQL. This is where the Lambdas come in. The Kinesis streams trigger the lambda, and the Lambdas filter and process the record and save them to a database on AWS.

Yeah, it all depends on how much load or incoming records you expect. In our case, we use the pipeline to import inventory, say images from from our partners. So, we might have a few million images coming in many times a month. Lambda is super efficient in this case.