CloudWatch Is of the Devil, but I Must Use It

Re:CloudWatch Is of the Devil, but I Must Use It by djhworld on Wednesday October 31, 2018 @04:35PM (#18349086)

The UI for Cloudwatch Logs completely crumbles if you use it for Lambda functions, especially if you have a lot of executions.

We run ETL pipelines and have lambda as a component of that, maybe 250k-500k invocations a day.

If you want to search through the logs for a particular invocation at a particular time (e.g. when an error happened) the UI just cannot handle it.

I think part of the reason is Lambda creates a new "log stream" every time a new container is created, which I'm guessing is how they shard the logs. So you end up with hundreds/thousands of log streams within your lambda log group and the backend has to search them to find the particular search pattern you are requesting.

Most of the time we have to export the logs to S3, download them and do some manual grepping.

[ Parent | on HN ]

Re:CloudWatch Is of the Devil, but I Must Use It by niklasrde on Wednesday October 31, 2018 @06:12PM (#18349669)

We move our Logstreams to our ElastiCache cluster to make them somewhat usable - using another lambda which receives subscription filters that are attached to a log group.

The processor also periodically scans new log groups under the /aws/lambda/ prefix, and creates subscriptions for them if they don't exist already.

Speaking about, this may be a good candidate for a more in-depth blogpost

[ Parent | on HN ]

Re:CloudWatch Is of the Devil, but I Must Use It by shshhdhs on Wednesday October 31, 2018 @06:19PM (#18349706)

Yes, I’d love to see the blog post!

[ Parent | on HN ]

Re:CloudWatch Is of the Devil, but I Must Use It by chair6 on Wednesday October 31, 2018 @05:19PM (#18349370)

Yeah, don't use the web UI. An alternative to awslogs is saw, https://github.com/TylerBrock/saw. Me likey..

[ Parent | on HN ]

Re:CloudWatch Is of the Devil, but I Must Use It by zedpm on Wednesday October 31, 2018 @04:38PM (#18349120)

I'd recommend against searching via the web UI. I use awslogs[0] daily and find it an effective way to search and filter logs.

[0] https://github.com/jorgebastida/awslogs

[ Parent | on HN ]

Re:CloudWatch Is of the Devil, but I Must Use It by scarface74 on Wednesday October 31, 2018 @05:15PM (#18349344)

For Python I use WatchTower to specify the name of the log group/log stream.

For C#, I use Serilog to specify the log group and stream. Another benefit of using Serilog, is that you get structured JSON logging with name/value pairs that you can use the JSON filters in cloudwatch abd you can send them to a NoSQL data store like Mongo and ElasticSearch and do more advanced queries than just regex expressions.

[ Parent | on HN ]