My worry here is that, in posting what seems to be a book, people just won't even do it because we don't have time to do it, unless this is a primary part of their jobs.

If a 10 minute guide gets users 90% of the way, then they're more likely to do it. And that's good enough to cover a majority of automated attacks.

Update: I take it back – they've provided scripts to run this stuff. I will explore these. Thanks for the link.

Back in the day, The Linux Documentation Project had a trove of hundreds of HOWTOs covering every facet of using Linux. By today's standards they seem like "books", but in reality they were step-by-step instructions for anything you could ever want to do in Linux. No digging through forums, no combing through man pages, no following broken outdated blog posts that didn't explain what you were doing. I find it sad that these are seen as detrimental today.

I feel like arch wiki is that thing today. It's a bit disorganised and things get outdated while nobody's looking, but there's almost anything you'd every want in there.

It seems like there a bunch of articles there but they're more like notes on how to use a tool specifically with Arch, rather than a guide for everyone.

Compare this page http://www.tldp.org/HOWTO/Quota.html to this page https://wiki.archlinux.org/index.php/Disk_quota . TLDP organized HOWTOs like mini-books; tables of content, multiple authors, versioned releases, and of course, you could download them all and search through them by category. And they didn't assume things like the distribution setting up a bunch of the tools and system for you, so you learned how the tools actually worked.

Sure, it's a different format. But you can always dig into man pages or other documentation to learn the details. Arch wiki is great for practical solutions - for example https://wiki.archlinux.org/index.php/HiDPI is the place to go to for hidpi display configuration. TLDP has not been seriously updated since hidpi even came out. Similar to https://wiki.archlinux.org/index.php/Power_management - there's no recent TLDP equivalent.

I appreciate TLDP for what it is and the details they go to. But Arch wiki can easily be read as a FAQ for all modern systems. And it will usually send you to other places for details (sometimes TLDP as well)

So are they outdated now?

A lot of the commands they documented have become obsolete or out of fashion.

E.g. Apache 1, BIND, sendmail, cgi-bin, ifconfig, etc.

Check out the arch linux wiki.

The problem isn't that it's long; the problem is that it's not navigable - there needs to be a table of contents. There's great stuff in here, but it's hard to sort out what I already know how to do from what I can actually use.

HTML version now has a TOC as well!

There is a table of contents in the PDF version.

Yeah, but it's a PDF. Anyway, why can't you generate a TOC into the HTML version from whatever format you're actually authoring in? Good tooling support should make this not hard.

EDIT: And now the HTML version has a TOC, too! Talk about immediate gratification -- kudos to whoever did that!

I wrote a tool the other day to help create a Table of Contents https://github.com/kaihendry/toc

Agreed - it is a bit long. But if you want to set something up from top to bottom it may be worth it. Cool - I saw the link to Github as well, looks like there is code to do much of what the article outlines. Awesome.

The hackthis application that is referenced in the guide is also in Github here:

https://github.com/inversoft/passport-js-example

It uses Ember, Node.js, Express, Sequelize, MySQL and Passport User Database (https://www.inversoft.com/products/user-database-sso).

Another not so long guide:

https://www.digitalocean.com/community/tutorials/7-security-...

>2048 bit RSA keys

Github recommends 4096 now, for what it's worth. [1]

>Pushing database backups offsite

This is a really bad idea and a good way to get owned. Database backups must be PULLED from the server, not pushed from it. Separately, you also need to test that you can restore from your backups periodically.

There were a couple other things I disagree with, but they're in the realm of personal preference. It's also interesting that this guide uses Linode which is a hosting company known to have had quite egregious security issues in the past. [2]

[1] https://help.github.com/articles/generating-a-new-ssh-key-an...

[2] https://www.google.com/#q=linode+hack

I agree that pulling does appear to be more secure, but if someone breaches any of your servers undetected, you're pretty much hosed regardless.

Plus, the Inversoft guide specifically states that backups must be encrypted. I could put my backup ZIPs on a public Github repository and no one would be able to access the user data stored inside it. Therefore, it really doesn't matted if they are pushed or pulled.

If a script or cronjob or whatever on the server has write access to the repo, so does the hypothetical attacker.

I remember seeing a "hacking" website get wiped along with all of its backups about 15 years ago and it left a very strong impression on me regarding this issue.

If it is write access but not delete access (nor overwrite access) then there is no issue.

It is sometimes more secure to push because it requires no inbound connections or authorizations the live machine.

That may be true but with a pulled db backup and a good recovery strategy you'll be back up relatively quickly, without you'll be DOA.

>This is a really bad idea and a good way to get owned. Database backups must be PULLED

Just make sure your database server doesn't have permission to delete backups (e.g. Have it POST a backup via HTTPS). There is nothing wrong with db server initiated backups.

This is still open to abuse in the form of your legitimate backup script being disabled and replaced by one uploading false backups. You wouldn't know anything has happened until you go to restore from one.

Of course, this is an edge case, but I believe setting up a pull-based backup system is still going to be less work than a write-only push system.

A hacked server can have the backup server pull false backups as well.

Of course, your servers shouldn't be SSHing to your backup servers, but that goes both ways.

If the backup machine is compromised the attacker can corrupt your backups. Either way, if machine is hacked you risk loss.

That's ridiculous. The same tricks can be played on a pull based system once you have compromised the DB server.

Why must they be PULLED?

If a server can push backups, it must necessarily have credentials for and write access to that backup repository. If the server is compromised, that write access could be used to attack the backup repository as well. Pulling backups from a server removes the need for credentials or access to the backup repository.

That is one valid way of solving this problem. Another option is to have a permission system that allows you specify that the server can only append new data, but not delete or overwrite existing files. This is useful if you don't want the added complexity of having a separate "backup management" service, which could easily become a bottleneck if you are backing up a lot of data from multiple servers.

One way to do this is with S3, for example, is to use an IAM role with only the "PutObject" permission, and enable object versioning for the bucket to prevent a compromised server from being able to delete data by overwriting existing files.

It's also easy to do this with Tarsnap. You can split out your host keys with tarsnap-keymgmt into separate read, write, and delete keys. Just take your delete key off of the host and store it somewhere secure.

Or better yet, create the key somewhere else and only upload the keys you want to have on the server. Otherwise there's a chance that the original tarsnap key is left in an unallocated block.

Not an expert here, but this seems to me to be stripping away one layer of security. Kind of like saying that you can leave a key to your house lying around as long as the pantry door is locked - if your database server gets owned, they now have some kind of login to your backup server as opposed to no kind of login.

Maybe the extra layer doesn't add much security, but if it's a simple config change and it does add something, wouldn't it be worth doing?

Not a bad point - it's a trade-off.

You could make a similar point about the centralized backup management server - it needs to have "some kind of login" to all your production database server, so of that host is compromised (which might only store encrypted copies of your backup), so are all your database servers if those privileges can be escalated. You could argue that one backup host is easier to secure than a complex system such as AWS, but then I would argue that it would probably be hard to beat the track record of S3/IAM. ;-)

Both approaches have their place. If you're dealing with a large number of hosts and a lot of data, the backup host will quickly become a bottleneck and you're probably better off with the append-only approach. If that's not a concern for you (i.e. you're not running into any bandwidth limits on the backup hosts) and want to deal with operating yet another service to avoid the risk of e.g. IAM privilege escalation, the other solution might be a better fit.

Doesn't entirely fix the problem, but using public/private key auth and a private key created just for the backup cron job, the server can push a backup via scp/ssh to its own user account which on the destination end is chrooted into a particular backup destination directory.

Plus, if you're super paranoid, pulling doesn't require leaving any ports open and software listening on the repository.

But it requires leaving a port open on the DB server where an attacker can get a whole copy of your database, or destroy it.

Many database servers are not serving databases solely to themselves and have that port open as part of the normal course of operation.

Could you recommend another hosting service with better security practices than Linode?

I don't currently use them, but I've had good luck with Vultr and they haven't had any incidents like Linode AFAIK. There's also AWS, Azure, whatever Google's thing is called, and all the other big ones. I have no opinion on DigitalOcean even though I currently have a VPS with them since I'm not running anything remotely mission critical there.

I don't know enough about server security to critique most of that guide, but the Password Security section is clearly behind the times:

> Passwords should always be hashed using a strong, one-way hash algorithm. [...] hashed with an algorithm like SHA-256 7 times.

If they had simply written "just use bcrypt" they'd be much closer to 2016-era security practices.

BCrypt works fine, but I wouldn't say it is "2016-era security practices". It was written in 1999 and hasn't had as much scrutiny as SHA or Blowfish (although it is based on Blowfish).

Regardless, using a salted, multi-pass algorithm will keep everything nicely secured using nearly any hashing algorithm.

Remember the goal is not to crack one user's password using a brute force lookup table, it is to crack everyone's password.

BCrypt is apparently just fine in 2016: https://news.ycombinator.com/item?id=11120528.

It is in fact a 2016-era good security practice.

And it is designed to prevent what your last sentence says.

We are preparing one focussing on application development:

https://github.com/FallibleInc/security-guide-for-developers... (work in progress)

Great checklist.

Yes. We originally just put this together just as a primer for younger engineers to understand the basic steps. The guide you posted looks really good though. It looks like it covers some of the application side of things as well.

So many people get some of the most basic security wrong. It is great to see people trying to educate instead of just pointing out issues. Thanks for this work.

I love the fail2ban part of your guide! I totally need to try that.

That is great as well. Quite extensive.

Great guide!