Even the word "small" is quite relative. One of my clients is big enough that you've probably heard of them, and their website/app (which is the company's only thing) is one LAMP server. That was only last year upgraded from a Windows 2003 WAMP server on a desktop. I tell friends what I did there and they say "wow you must be really good at Puppet at scale" based on assumptions.

Most people have yet to discover what a tiny little machine can do if you spend some time addressing bottlenecks. I have fond memories of running websites on the cheapest VPSs I could find, and getting very respectable performance from them with little more than cutting down on resource-hungry services, going static html whenever possible and being vigilant about any resources loaded by the browser.

See also https://lowendbox.com/ and https://hackernoon.com/10-things-i-learned-making-the-fastes...

I run a couple of very small scale (mostly static content) web sites and E-mail serving and a few other services on a single "Cheapest VPS" and have never gotten interested in Puppet or Ansible or any of the other things mentioned in this thread. It just seems like adding abstraction and automation on top of things that are not worth abstracting or automating. To me, these things are Yet Another Software That Could Fail. When I need to change a configuration file, I ssh in, sudo vi /etc/blahblah.conf and get on with my life.

I just use versioned Makefiles for my very simple personal projects with my artifacts and configs in git. Pull the repo to the box, and run the make file. I commit config file changes then pull and make as needed. Everything is simple, it's manual automation so there is nothing running, but it's repeatable.

I put the bootstrap into an OS package and my server images point to my personal repo so I can configure run and build time dependencies as needed. At the end of the day, I can get a new server, run update, install my package and walk away and everything should be running, but it takes almost no additional overhead over doing it all by hand the first time.

I also was never interested in Puppet, however I was only interested in Ansible, and only use it as sort of an automated check list when setting up my server, in case I need to rebuild the server, the tedious steps can be automated. But I've never used it for multiple server management.

So what happens when the hard drive fails and you lose all the data and configuration on the box?

Avoiding the huge, fast-changing software for configuration management doesn't mean you can't have CM or high availability on cheap boxes. The OpenVMS approach (see Section VI) combined good filesystem, clustering, optional OS-level virtualization, and distributed lock manager to have clustered systems that ran for years without downtime. Record was 17.

http://h41379.www4.hpe.com/openvms/whitepapers/high_avail.ht...

Those few building blocks done in a highly-robust way on one of the stable, Linux platforms could probably achieve the same thing. Just RAID 0, a good filesystem, clustering itself, and backups would get far. I know there's Linux-oriented products out there but I don't know if their availability or failover time have caught up yet.

My VPS host fixes it and I restore from backup.

That’s what RAID and daily backups are for.

Yep. I got to say to my boss recently "you know those 2-4 servers I requested? You can forgot about that now, I optimized some things and we have plenty of capacity now on the one machine we've been using." That's an easy conversation to have.

I share your viewpoint and am currently thinking through a similiar small-scale deployment. How do you handle logging? For single boxes I'd just use logwatch and friends, but for aggregating log output (both system/application logs) from a small-ish Consul cluster I feel the options are complete overkill. I've had a look at ELK, time-series backed stuff (e.g. Prometheus), but all I really want is log aggregation with search capability and optionally Regexp-based alerting.

It seems to me Logstash provides what I want, but I sure as hell won't run 3 JVMs and a Redis instance to aggregate logs.

tl;dr How to handle log aggregation within a small-scale cluster without losing your sanity?

Monitoring and logging are best separated. Prometheus is great even on a small scale. For log aggregation there is the relatively new oklog.

I have been running oklog for a customer successfully. You ought to deploy it like a regular (non-cloud-native) service though: When a server crashes, restore the oklog instance from backups, not spinning a new one.

For small stuff I still like the simple syslog. It’s usually built in and supports aggregating to a single server.

In the past have created a server just with syslog. Adjust all your servers syslog configs to point to that one. Then log in and use grep.

Not fancy. But gets the job done with a single line config change in your syslog configs.

Senior sysadmin here, and I agree this is one of the most common methods. I have also had great success tying in the ossec or pam module notifications into syslog messages that all go to the central syslog (nsyslog/rsyslog etc) server. The problem with the elk and similar stacks imho is lack of security and speed. Things this old school setup doesn't have a problem with. The problem is that too many managers don't like not having gui dashboards... which is why splunk et al have really taken off.

This re-enforces my idea that a purely terminal based business dashboard might be a cool product with a fairly large market. I've been eyeballing some of the go/ncurses work for this.

rsyslog or syslog-ng were preferred these days for better security and performance I thought? Unless you’re using something like stunnel to wrap the syslog messages that is.

If you’re not violently opposed to MongoDB Graylog is very popular. I haven’t heard bad things about it so far besides the MongoDB portion but log aggregation is one of those systems that may be ok with some hiccups from time to time as opposed to an actual primary business data store.

We're kinda big on Splunk, everything that remotely looks like log data get shipped of to Splunk.

I've worked with ELK before and I'm not a fan, the usability is very low compared to Splunk. It's much cheaper though. At a previous employer we switch from Splunk to ELK and the number of searches we did on a daily basis dropped to almost zero. Before that almost every support "ticket" would start with a Splunk search.

You could also try out https://www.humio.com. I only seen it deploy by one customer, but it looks nice.

We use Sumologic and it’s really nice. Much better than ELK. I’ve only used splunk in a evaluation context a long time ago, so I’m not sure how they’d compare. Splunk is just so expensive.

I found Sumologic to lack features that ELK has such as packetbeats. But you still have to manage the Sumologic forwarding agent and keep under your ingest limit. Additionally my mind never exactly flowed with their query syntax. They refreshed the UI recently and I liked it (aside from compounding my overutlization of tabs haha), you now more easily/graphically select time series from the graph. I can see how it is nice to not have to worry about re-indexing and what type of device the underlying data rests on. I have not tried the ELK hosted solution so some of my criticisms could apply to it as well.

We use Humio alongside Splunk and they have an office in SF. Nice people, price is nice too :)

I just have a server dedicated to syslog and have rsyslog running on everything else.

For simple, non-critical stuff this is the simplest solution I've come up with - if you don't need real-time aggregated logs (i.e. you only want to archive them) and you have log rotation configured, you can simply use cron with "aws s3 sync" or rsync for the logs folder.

Check out papertrail.