You have a lot of decent options. You could do a lot worse than ELK. If you're on AWS, you can get hosted Elasticsearch. It comes out of the box with Logstash you can hook up to DynamoDB, and it also does Kibana out of the box. There are a number of other vendors; but there are decent reasons for keeping your logs as close as possible.

CloudWatch works fine too. CloudWatch comes integrated with AWS services out of the box. It can be more annoying to get your logs into it than ELK (the latter seems overall more popular). Its alerting and the AWS CLI integration pretty slick, though.

You should also go turn on CloudTrail right now. It lets you automatically log side-effectful API calls. It is not a replacement for a centralized logging pipeline, but it's great high-signal data to put into one.

I appreciate that your complaint (totally valid!) was "this is a rabbit hole", and I just gave you two options, and that might not help your perception that it's a rabbit hole. If you find yourself paralyzed by choice, either choice is much better than deferring the choice! Just pick one. Heck, if you can't pick, let me help: pick AWS hosted Elasticsearch.

A lot of people (also in the security space) like Splunk. I find it annoying to deploy (I've heard rsyslog-in-front-of-forwarders as a canonical deployment method for just ingesting syslog more than once because reasons) and overpriced. YMMV.

Disclaimer: shameless plug! You're not the only one with your hair on fire. One of the first things we're doing for Latacora customers is setting up a centralized logging pipeline.

I second ELK, and I even stronger-second Splunk being overpriced (with the caveat that if you do deploy it, I think it's the best option, just not really worth it).

I think it's really important to internalize the idea that there is no Platonic ideal of a logging solution. It's a fundamentally frustrating manifestation of entropy that you're going to wrestle with, but it's a really necessary goal to work towards long term. Sort of a "the first step is admitting powerlessness" kind of deal.

I've had good luck with Cloudwatch and, if you're on AWS, I'd recommend it over any other hosted log system (with the possible suggestion of a more elaborate ELK setup that you build yourself).

The trick to Cloudwatch is --- like most AWS services --- never using the web UI.

That's a good point! If you have someone consuming it that wants a (shared) web UI, you want Kibana. If they prefer to consume their text in a terminal and are fine with typing `aws logs` a bunch, CloudWatch is fine (and probably a little less twiddly than ELK).

These are such great comments, thanks for sharing your insights. For folks looking for other options, I'd also mention https://honeycomb.io, perhaps the most promising newcomer in this space. It's essentially Facebook's Scuba for the rest of us.

It depends a LOT on how many machines and services on the machines you're dealing with. There's a remarkable amount of stuff at the small end which is good, cheap, and fast to deploy.

I've been using Loggly for my personal machines (~8, mostly cloud VPSes). On the plus side, it's free at my scale, and the analysis and reporting tools are nice at least in theory. On the minus side, I can't get my logs past 7 days archived to S3 without paying $150/month, which I really want since my main use-case is longer-term analysis and forensics.

I'm planning to switch to Papertrail, which for the princely sum of $7/mo will give me a simpler UI and a year's archiving to S3.

Loggly and Papertrail both use the same deployment strategy (you hook them up to syslog and/or your app's logging package), and I had Loggly up and running and providing useful feedback in solidly under four hours.

For small to medium log volume, I can only recommend Loggly.

The killer feature it has is for me is searching structured (JSON) logs. Just use the Logstash/Greylog library in the language of your choice and send the logs to Loggly, and you quickly have a logging system where you can zoom in on the logs comming from different subsystems of your codebase or produced by a specific user.

I would use some SaaS solution. ELK can be cheaper, but can take tons of time to configure and maintain if you run it. For early startup paying under $100 / month for some logging solution is no brainer vs. spending time configuring.

Disclaimer: I work at Sumo Logic I would recommend: https://www.sumologic.com On top of grep like searches, you can do analytical searches (SQL on text data).

Here are some SaaS choices:

* Sentry: https://sentry.io/welcome/

* Logentries: https://logentries.com/

* Loggly: https://www.loggly.com/

* Opbeat: https://opbeat.com/

* Papertrail: https://papertrailapp.com/

Sentry is open source and there is even an official up-to-date docker image: https://hub.docker.com/_/sentry/

Loggly published an "Ultimate Guide to Logging": https://www.loggly.com/ultimate-guide/

Please excuse the shameless plug but since you are asking for one, Striim is a good out of the box centralized logging solution. We use Kafka as our messaging layer (you can either install on your own Kafka or use our internal) and we use Elasticsearch as the storage layer.

We also have streaming log parsers to connect your data. That whole thing about 'creating new alerts in minutes' is trivial in our platform since everything is based in SQL.

Unlike Splunk or ELK, our solution is based on in-memory streams so you don't have to wait for data to be indexed to fire off alerts on anomalous activity. Feel free to message me to find out more or simply download the product from http://www.striim.com/

TFA aside, centralized logging is super useful for debugging a variety of issues. There are a number of hosted options, and setting them up isn't too hard. It usually involves configuring you're application's log device to talk to the remote service, or configuring syslog on your app servers to forward logs to said service.

See https://logentries.com/ for an example

If you want a pretty prepackaged solution you could do a lot worse than splunk. They even offer it SaaS

https://www.splunk.com/en_us/cloud.html

I'm in the same boat. Looking for recommendations on strong, sturdy, buckets, for bailing water.