News for hackers. Stuff that matters.
Re:Show HN: Serverless logging to S3 by apexkid on Tuesday November 07, 2017 @05:46AM (#15642333)
Why not go for App -> AWS Firehose -> S3 architecture. Firehose will automatically take care of batching logs and putting them together. Also it can create partitions. No need to use Lambda at all.
Re:Show HN: Serverless logging to S3 by guitarbill on Tuesday November 07, 2017 @06:13AM (#15642439)
Possibly because the app is running on Lambda? Cloudwatch logging is not great for Lambdas.
Re:Show HN: Serverless logging to S3 by szalonypiekarz on Tuesday November 07, 2017 @06:40AM (#15642544)
In that case, why not Lambda -> Cloudwatch Logs -> Firehose -> S3 ?

No bespoke logging solution is needed.

Re:Show HN: Serverless logging to S3 by joemaller1 on Tuesday November 07, 2017 @06:54AM (#15642611)
Don’t assume the author made an informed but poor choice, they likely made something work with the tools they know or found. It’s really difficult to keep up with all the various AWS offerings if your not part of a team that’s immersed in it. Constructively suggesting alternates is welcome, I’ve got some new tools to read up on this morning.
Re:Show HN: Serverless logging to S3 by throwaway2016a on Tuesday November 07, 2017 @12:14PM (#15645239)
> Cloudwatch logging is not great for Lambdas.

I run some Lambda systems at scale and CloudWatch Logs has worked excellent for me. It was completely painless and CloudWatch logs has a lot of tooling that comes with it (backups, streaming, indexing with Elastic Search, etc).

What part of it has not worked well for you?

Re:Show HN: Serverless logging to S3 by guitarbill on Tuesday November 07, 2017 @04:24PM (#15647793)
Mainly searching and exporting. Exporting logs to S3 manually for debugging/post-mortem is harder than it needs to be - maybe this isn't a huge use-case, but we like to attach logs to tickets.

And searching is very limited. Yes, you can stream them to ElasticSearch => LogStash/Kibana, etc but that's extra steps, and I can do similar things completely without CloudWatch Logs, e.g. Sentry and SumoLogic integrate fine with Lambdas also.

I'm not hating on CloudWatch Logs, and alarms are good. It's just not a fully featured logging solution, and it sounds like you agree with that?

Re:Show HN: Serverless logging to S3 by throwaway2016a on Wednesday November 08, 2017 @09:31AM (#15652711)
I agree you need more tools around it to make it a full logging solution. Doing data visualization is more or less impossible without extra tools. But I've overall liked it.

Search is powerful enough to do something like say, "get me all messages with this trace GUID" and S3 exports can be automated[1] and you could probably use the "task-name" to associate with your ticketing system.

[1] http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3Ex...

Re:Show HN: Serverless logging to S3 by jcims on Tuesday November 07, 2017 @10:38AM (#15644240)
Why is that?
Re:Show HN: Serverless logging to S3 by ellimilial on Tuesday November 07, 2017 @07:44AM (#15642874)
Thank you for the feedback.

I was considering Kinesis but my use case (multiple small apps not running on AWS sending medium-sized messages a few times per user session) values simple components and keeping the long-term maintenance effort to the minimum.

Re:Show HN: Serverless logging to S3 by weego on Tuesday November 07, 2017 @08:34AM (#15643200)
I'm not sure what your understanding of Kinesis Firehose is but, in this situation it would literally remove some moving parts, it would not add complexity. Also potentially cheaper than SQS anyway.
Re:Show HN: Serverless logging to S3 by ellimilial on Tuesday November 07, 2017 @09:20AM (#15643519)
Yup, makes sense, looks like it's a decomposition of it. Might have been the talk of installing agents, sharding etc that put me off track.