In addition, it limits available commands to a certain predefined subset, allowing the host to prevent damage caused (e.g. a DoS attack in this case) by the system being compromised.

I ran kippo for a while and it seemed that all attackers were trying to upload files over SCP, which kippo does not support. A few attackers resorted to logging in and downloading with wget. However, the vast majority of attacks ended with a failed SCP session.

Check it out here: https://github.com/micheloosterhof/kippo

The typical intruder in our case uses wget to pull down files and that works w/o a hitch.

If you have the time to run it AND check on it you'll learn a lot.

Not to say that kippo isn't good, I didn't look too closely but it seems to be mainly focused on ssh and terminal capture.

Would it have recorded also statistics like the connection activity?

Seeing all the UDP traffic, and being able to trace its origin to the "@udp1 39.115.244.150 800 300" command, received not via shell but via a TCP connection, was pretty cool.

Rather, his server connected to some IRC server and joined a channel with all his other bot friends. The owner then sent the command to the IRC channel, and it is then broadcasted to all bots by whatever IRC server he is using.

(This is why lots of IaaS providers will forbid you from hosting an IRC server, and sometimes block all IRC traffic (by port anyway) on their networks)

In addition, there are quite a few good visualization tools to show the logs made by kippo. You can save them to a db, plot them nicely, etc.

Based on the timestamps of the entered commands, I guess one of the takeaways for the attacker is to look into config management tools (eg ansible) :)

[1]https://news.ycombinator.com/item?id=7622121

That's terrible!

AFAIK, AWS defaults to ssh-key logins with password logins disabled. Can someone comment about Rackspace/DO?

$ uptime

  22:09:07 up 30 days, 12:19,  2 users,  load average: 0,17, 0,09, 0,07

Password:

Status for the jail: ssh-iptables

  |- filter
  |  |- File list:	/var/log/messages 
  |  |- Currently failed:	1
  |  `- Total failed:	1757
  `- action
     |- Currently banned:	0
     |  `- IP list:	
     `- Total banned:	242

    up 298 days, 20:42,  1 user,  load average: 0.00, 0.01, 0.05

The distinction is, though, that the SSHd on that box is running on a non-standard port (220)... so that certainly makes a difference.

1) Password authentication 2) Root authentication 3) Changed root password to "password"

All the providers offer fairly safe defaults, either using very random passwords or just enabling SSH keys.

It is good to know that all providers have safe defaults, I only have experience with AWS in that regard.

As far as I know, you can still create/publish AMIs where password auth is enabled, but all of Amazon's stock images only allow ssh-key auth.

Having said that, it's amazing what OP could do with just one monitoring tool. Very impressive.

Wouldn't it have been better if the attacker had removed only the last few lines recording his commands from the log files instead of the entire files? Wouldn't the lack of continuity in the log files be very noticeable?

Also, is this a script running this sequence of commands or an actual person?

And, is there a log somewhere on the system of 'make' activity?

1) Yes, it would have been better but I honestly think this attack was completely botnet-driven and the attacker didn't really mean to cover his footprints too much: in the timespan of 10 minutes, he sent over 800 MB of UDP traffic. That would have been caught even by the most oblivious sysadmin pretty quickly, so these guys are just playing a number game, trying to break in as many hosts as they can knowing that the lifespan of the hacked hosts will be very short, maximizing the short-term profit then.

2) The attacker directly ran these commands on the login shell (no script was copied over scp or something else), so there was no script executed on the host itself, but the whole thing lasted roughly 2 minutes and a lot of commands were "typed", so I am almost sure this was just an automated script ran from another probably compromised host.

3) I didn't check if the build left logs, but by showing every executed process with "evt.type=execve" (which goes deeper than the spy_users chisel) you can see all the processes executed by the build: 99% are just uninteresting sed/gcc/autoconf.

Kernelwise, the main thing to report is a couple of crashes on non-mainline kernels like openVZ.

I found chippy's comment interesting and helpful and don't know why it was downvoted to hell while other "+1"-style comments (that didn't add any value) are left as-is: https://twitter.com/taoeffect/status/464090445677481985

I was able to find all the attempts by looking at the I/O activity of the sshd process, and also the syslog activity recorded every attempt.

Other providers (such as Digital Ocean) use the root account by default even for Ubuntu, although the password is set to a really secure and random one.