News for hackers. Stuff that matters.
Re:Show HN: Serverless logging to S3 by apexkid on Tuesday November 07, 2017 @05:46AM (#15642333)
Why not go for App -> AWS Firehose -> S3 architecture. Firehose will automatically take care of batching logs and putting them together. Also it can create partitions. No need to use Lambda at all.
Re:Show HN: Serverless logging to S3 by guitarbill on Tuesday November 07, 2017 @06:13AM (#15642439)
Possibly because the app is running on Lambda? Cloudwatch logging is not great for Lambdas.
Re:Show HN: Serverless logging to S3 by szalonypiekarz on Tuesday November 07, 2017 @06:40AM (#15642544)
In that case, why not Lambda -> Cloudwatch Logs -> Firehose -> S3 ?

No bespoke logging solution is needed.

Re:Show HN: Serverless logging to S3 by joemaller1 on Tuesday November 07, 2017 @06:54AM (#15642611)
Don’t assume the author made an informed but poor choice, they likely made something work with the tools they know or found. It’s really difficult to keep up with all the various AWS offerings if your not part of a team that’s immersed in it. Constructively suggesting alternates is welcome, I’ve got some new tools to read up on this morning.
Re:Show HN: Serverless logging to S3 by throwaway2016a on Tuesday November 07, 2017 @12:14PM (#15645239)
> Cloudwatch logging is not great for Lambdas.

I run some Lambda systems at scale and CloudWatch Logs has worked excellent for me. It was completely painless and CloudWatch logs has a lot of tooling that comes with it (backups, streaming, indexing with Elastic Search, etc).

What part of it has not worked well for you?

Re:Show HN: Serverless logging to S3 by guitarbill on Tuesday November 07, 2017 @04:24PM (#15647793)
Mainly searching and exporting. Exporting logs to S3 manually for debugging/post-mortem is harder than it needs to be - maybe this isn't a huge use-case, but we like to attach logs to tickets.

And searching is very limited. Yes, you can stream them to ElasticSearch => LogStash/Kibana, etc but that's extra steps, and I can do similar things completely without CloudWatch Logs, e.g. Sentry and SumoLogic integrate fine with Lambdas also.

I'm not hating on CloudWatch Logs, and alarms are good. It's just not a fully featured logging solution, and it sounds like you agree with that?

Re:Show HN: Serverless logging to S3 by throwaway2016a on Wednesday November 08, 2017 @09:31AM (#15652711)
I agree you need more tools around it to make it a full logging solution. Doing data visualization is more or less impossible without extra tools. But I've overall liked it.

Search is powerful enough to do something like say, "get me all messages with this trace GUID" and S3 exports can be automated[1] and you could probably use the "task-name" to associate with your ticketing system.

[1] http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3Ex...

Re:Show HN: Serverless logging to S3 by jcims on Tuesday November 07, 2017 @10:38AM (#15644240)
Why is that?
Re:Show HN: Serverless logging to S3 by ellimilial on Tuesday November 07, 2017 @07:44AM (#15642874)
Thank you for the feedback.

I was considering Kinesis but my use case (multiple small apps not running on AWS sending medium-sized messages a few times per user session) values simple components and keeping the long-term maintenance effort to the minimum.

Re:Show HN: Serverless logging to S3 by weego on Tuesday November 07, 2017 @08:34AM (#15643200)
I'm not sure what your understanding of Kinesis Firehose is but, in this situation it would literally remove some moving parts, it would not add complexity. Also potentially cheaper than SQS anyway.
Re:Show HN: Serverless logging to S3 by ellimilial on Tuesday November 07, 2017 @09:20AM (#15643519)
Yup, makes sense, looks like it's a decomposition of it. Might have been the talk of installing agents, sharding etc that put me off track.
Re:Show HN: Serverless logging to S3 by CSDude on Tuesday November 07, 2017 @05:52AM (#15642362)
I am not trying to be rude but SQS is not meant for this kind of usage, Firehose to S3 is much more useful.
Re:Show HN: Serverless logging to S3 by ellimilial on Tuesday November 07, 2017 @07:35AM (#15642830)
Sure, Kinesis is a clear candidate for a larger scale scenario, hence all the disclaimers.

This is aimed at small-scale operations, utilising hopefully simple components.

Re:Show HN: Serverless logging to S3 by leevlad on Tuesday November 07, 2017 @10:33AM (#15644180)
Kinesis works well for low volume, and isn't any more or less complicated than SQS. You should give it a try, it will certainly make some things simpler
Re:Show HN: Serverless logging to S3 by jamiesonbecker on Tuesday November 07, 2017 @10:45AM (#15644319)
Even though it'd work, kinesis would probably not fit this use case very well (sporadic/low volume logging), since you have to pay every hour, whether you're using it (or barely using it) or not.

Firehose would work, though... no per-hour charges.

Re:Show HN: Serverless logging to S3 by opmac on Tuesday November 07, 2017 @09:51AM (#15643773)
Firehose works for any scale, small or large.
Re:Show HN: Serverless logging to S3 by mooreds on Tuesday November 07, 2017 @08:33AM (#15643183)
Cool idea. I love that you're very clear where this makes sense and where it doesn't. For higher volume logs files I'd definitely consider something like kinesis agent: https://github.com/awslabs/amazon-kinesis-agent

But for lower volume logging this seems like a cost effective solution, and then you can use Athena or EMR to further process or analyze the logs. I didn't see any partitioning of the logs by date--that might be a nice enhancement (unless I missed it).

Re:Show HN: Serverless logging to S3 by haydenkshaw on Tuesday November 07, 2017 @06:07AM (#15642420)
Would probably use CloudWatch Logs for this personally
Re:Show HN: Serverless logging to S3 by alex_duf on Tuesday November 07, 2017 @09:21AM (#15643538)
Yes if the goal is to archive stuff, event per event I'd use cloudwatch logs. You can create the stream you want in order to log to it.

It's probably cheaper but lambdas and s3 are equally dirt cheap so hey why not.

Edit: Two possibilities here. Either the choice of technology is deliberate and there's a reason I can't quite figure out, or AWS has reached that state where they do so much it's really really hard for people to know what service to use for each problem.

Which actually makes me wonder what am I doing right now that could be done easily using a pre-baked AWS solution?

Re:Show HN: Serverless logging to S3 by mooreds on Sunday November 12, 2017 @11:05AM (#15681263)
I teach some AWS courses and that's something I always bring up: if you have a system you are thinking of building it always makes sense to see if AWS has a prebaked solution and evaluate it. The solution may not meet your needs but is likely to be available at a price/performance ratio (especially considering fully weighted operational costs) that will make it compelling if it does meet the requirements.
Re:Show HN: Serverless logging to S3 by ellimilial on Tuesday November 07, 2017 @07:52AM (#15642912)
It sounds like a much easier solution for EC2 instances. However it's limited to AWS - hosted apps, if I'm not mistaken?
Re:Show HN: Serverless logging to S3 by code4tee on Tuesday November 07, 2017 @08:25AM (#15643121)
No, you can use cloudwatch anywhere you can install the AWS CLI—-including non AWS assets. Just point the log monitor at a file or folder with the right CLI commands and you can suck up logs in real time into cloudwatch.
Re:Show HN: Serverless logging to S3 by loevborg on Tuesday November 07, 2017 @09:22AM (#15643550)
Can you tail cloudwatch logs in real time, ideally from the command line?

For debugging situations, I'd like to send a request to my app server and then see the server response live, or with little delay.

Re:Show HN: Serverless logging to S3 by zedpm on Tuesday November 07, 2017 @10:45AM (#15644311)
Yes, I use it in just that way. I recommend installing awslogs[0] on your workstation. Then, to tail a log group, you'd run

   awslogs get <group_name> -w
The latency is pretty low and this generally works well. You can apply filters etc. as well if needed.

[0]: https://github.com/jorgebastida/awslogs

Re:Show HN: Serverless logging to S3 by cosenal on Tuesday November 07, 2017 @05:18AM (#15642191)
This should have the «Show HN» header
Re:Show HN: Serverless logging to S3 by ellimilial on Tuesday November 07, 2017 @05:24AM (#15642217)
Done, thanks for the pointer.