When I'm on-call and get an alarm on a server, checking that colleagues aren't logged in is normally the first thing you do. If a service fails it's more often than not someone doing maintenance, and they just forgot to tell you, or disable monitoring. And as someone else pointed out, also check disk usage.
Hacker News can be a little blind to the fact that most software projects are rather small and modern servers are really powerful. It's actually pretty rare that someone manages enough infrastructure for a single service that logging isn't a viable option.
See also https://lowendbox.com/ and https://hackernoon.com/10-things-i-learned-making-the-fastes...
I put the bootstrap into an OS package and my server images point to my personal repo so I can configure run and build time dependencies as needed. At the end of the day, I can get a new server, run update, install my package and walk away and everything should be running, but it takes almost no additional overhead over doing it all by hand the first time.
http://h41379.www4.hpe.com/openvms/whitepapers/high_avail.ht...
Those few building blocks done in a highly-robust way on one of the stable, Linux platforms could probably achieve the same thing. Just RAID 0, a good filesystem, clustering itself, and backups would get far. I know there's Linux-oriented products out there but I don't know if their availability or failover time have caught up yet.
It seems to me Logstash provides what I want, but I sure as hell won't run 3 JVMs and a Redis instance to aggregate logs.
tl;dr How to handle log aggregation within a small-scale cluster without losing your sanity?
I have been running oklog for a customer successfully. You ought to deploy it like a regular (non-cloud-native) service though: When a server crashes, restore the oklog instance from backups, not spinning a new one.
In the past have created a server just with syslog. Adjust all your servers syslog configs to point to that one. Then log in and use grep.
Not fancy. But gets the job done with a single line config change in your syslog configs.
This re-enforces my idea that a purely terminal based business dashboard might be a cool product with a fairly large market. I've been eyeballing some of the go/ncurses work for this.
I've worked with ELK before and I'm not a fan, the usability is very low compared to Splunk. It's much cheaper though. At a previous employer we switch from Splunk to ELK and the number of searches we did on a daily basis dropped to almost zero. Before that almost every support "ticket" would start with a Splunk search.
You could also try out https://www.humio.com. I only seen it deploy by one customer, but it looks nice.
For my startup, I often run bash on Heroku because I do migrations manually (again, it's a feature, I'm too inexperienced to have automated migrations that work everytime, I prefer to be already on it if it breaks). Sometime when something breaks I'll also poke around the filesystem (which is a copy, so no fear to break anything).
Basically, I'd say the smaller your team, your uptime requirements and your traffic is, the less you need automation, and the more you are susceptible to login directly to a box (I combine all of that: team of 1, no uptime requirement, not enough traffic to even max the most basic server).
(Also, when someone asks me "how do you configure X", I can just link them to the corresponding place in my system configuration repo on Github.)
The reasons for this are several, but the most pressing are that if one falls over I can spin it up again quickly somewhere else, and I can have a lot of commonality in the configurations between them, so they all have backups configured the same way, or setting up letsencrypt requires editing things in only one place.
It also means I don't have to dig through a bunch of config when I want to work out how stuff is set up three years later, I can just look in one version-controlled directory in a central location.
In web scale production, you mostly have to worry about your fellow sysadmins troubleshooting the problem, and they mostly won't be too mad about you clearing state.
But not everything is web scale production. Productionizing an app is a lot of work. Yes, yes, "Cloud is reliable!" and that's kinda true if you write your application to deal with any failure at any time. the reality is that the hardware can and will sometimes go away without warning, and you can't call up your oncall and tell them to head to the datacenter to fix it. all that recovery is done at the application layer; you had better hope you managed to make the data redundant enough. The whole idea behind the "cloud" is to take most of the lower level sysadmin work and make developers do it; and that's a fine way to some things, but... not so fine when it comes to other things.
(This is the value proposition of a VPS system rather than a "cloud" instance. when the hardware a VPS is on goes down, some poor bastard's pager goes off, and they are expected to wake up, drag themselves to the datacenter and fix it. The gamble is "is being down for a few hours now and again, when you can reasonably expect to be brought back up in the same configuration cheaper than writing your software such that you can always start over on a new node?" The VPS is for the former, the "Cloud" for the latter.)
Add to that, well, a lot of businesses need to run proprietary software. I personally, well, let us just say that for the last three years, the systems I am dealing with at work involve FlexLM.
so corp and smaller sites are littered with one-off systems... systems where if they break, a pager goes off, and someone fixes the problem. Sometimes that works better than the "web scale" stuff... like I've yet to see a "web scale" posix-ish filesystem that meets the user expectations the way NFS does, and I've never seen a nfs server that didn't require someone on pager.
Ansible doesn't get you everywhere.
InfluxDB is a simple command to extract what I need and then work with it from there.
I also had a difficult time to explain that a prod deploy of 30min (image creation, deploy with blue green) is normal for this kind of inf... Did you face the same thing?
At the end of the day, if you're on continuous deployment, a commit should be rebuilding only what it touches. We have 4 min long deployments + 1.5 min tests and I definitely don't think we're optimizing aggressively.
Many places still keep producing very stateful software (sometimes even very much by choice) that is better off managed through Puppet / Chef rather than an immutable containerized approach. If your software needs to take an hour and a half to shutdown, for example, you have to get a bit creative with your deployment strategies.
But they aren't interested in investing in more infrastructure when a single LNMP server does the job.
I think there’s another level we need beyond treating servers as pets or cattle, which is treating servers as wild animals; after you’ve captured one and interacted with it, you’ve doomed it because now it has the scent of humans on it.
I haven't seen human users in passwd, since virtualisation kicked in couple of years back.
My experience of the last five years is so heavily weighted to (effectively) immutable infrastructure that checking to see who had been on a box hadn't event crossed my mind.