> You should treat the server as compromised and rebuild from metal, by the way. I know that is annoying as heck but they clearly got code execution and you can therefore assume they had root if they wanted it and that any attempts to detect whether they did are useless because their rootkit makes the box lie to you about its current state.

Is privilege escalation that easy/common? Thinking esp of the number of shared hosting providers out there, if a user account is compromised they don't assume the entire server is compromised.

Now ok, www-data isn't any old user account, but the same principle applies?

In theory you might be right, but in practice patio11 definitely is.

Either way, the big hassle is going to be reinstalling your site, pulling a copy of the database from backup (you have that right?). Might as well go all the way and install everything fresh.

Why reinstall database from old backup?

Can trojan be hiding in the database?

Yes, definitely! Drupal (like Wordpress and others) stores all the "content" of your site in a database so there could be all manner of nastiness hidden in your pages.

And then, obviously, the attacker could have added an admin user account or, less obviously, altered settings stored in the database to make the site insecure.

My advice would be to restore completely from backup, if possible.

The price of restoring from old backup could be steep: some data loss. Or at least time consuming manual data merge with data from more recent, but compromised database.

It might be easier to check database for common vulnerabilities (admin accounts, suspicious content for web pages).

If you must go down that road, I'd suggest pulling the content off the compromised site and copying it manually and very carefully to the recovered site. You can never really be sure you got rid of everything otherwise.

Yes, privilege escalation is common. If someone gets code execution on your server, assume they got everything.

TL;DR: If owned, start from a fresh base system.

I think you meant "someone gets privileged code execution," which is a sensible assumption. Even still, app-permission (less than privileged) code execution can still do damage like host malware, IRC dumpsites/bot control, diodes, tor relays, vandalize web properties, etc.

The only way to know that a system is no longer owned for certain is to reimage it to a known good state. Doing anything less is tons of work, and unlikely to catch everything (rootkits, backdoors, hidden services, replaced system files, etc.). Even when running HIDS, HIDS cant be trusted because rootkits can hide things from it because it's running from the system with a possibly infected kernel. So, it turns out reimaging is less work and more trustworthy if the box is rebuilt and the 'sploit can be mitigated before bringing it online to the outside world (build and patch offline to avoid getting re-owned).

While I agree in principle that when owned one should start from scratch, my advice would be to learn something. Often I am asked to analyse attacks and I now have a collection of about 15 malware scripts that not only show me the intent but they are also useful (and remarkably well coded) for my daily admin tasks.

So, fresh start but at least get something out of it!

"and the 'sploit can be mitigated before bringing it online to the outside world"

You should read more carefully.

Also, keeping people waiting without an ETA for a down service because you're learning isn't going to result in happy customers.

Furthermore, whomever is running these boxes needs to deploy NIDS and HIDS and properly secure their boxes, because clearly they don't understand what an attack surface is.

Yes, whatever your security problem is, I'm sure some NIDS will clear it right up.

What a flippant, uncivilized, unconstructive comment.

Defense in-depth, every little bit helps.

Disagree, both in spirit and to the letter; for starters, I'm pretty sure there's still validity in a very long blog post I wrote about NIDS back in 1998:

http://insecure.org/stf/secnet_ids/secnet_ids.html

People running SAAS apps probably shouldn't waste much time with NIDS.

I'd assume s/he would isolate that machine for post-mortem and spin up a new service on a different box, unless we're talking poverty IT.

I think he said it precisely as he meant it.

> Is privilege escalation that easy/common? Thinking esp of the number of shared hosting providers out there, if a user account is compromised they don't assume the entire server is compromised.

Privilege escalation is easier than getting the initial shell. I would certainly reinstall any machine where someone has a shell.

As for these web shells, this agin demonstrates the important of blocking outbound connections with a firewall or some type of networking design.

Shared hosting providers stay in business with the principle that somehow they provide a relatively secure platform for their users, so a few of them do harden their systems enough to prevent most priv escalation. But one buffer overflow and a lack of mandatory access control later and you've got gold. Priv escalation exploits have been so common in Linux in the past decade that I would consider local access the same as root access [without MAC like grsec or selinux enabled].

Out of interest, have you any opinion on selinux vs grsec vs apparmor, in terms of which is better in certain scenarios and/or overall?

Well apparmor is like the baby cousin to selinux, or the big brother to the traditional unix DAC. Selinux is more robust but also more complex. Grsec is an attempt at both a MAC and various exploit-prevention techniques. In that sense, grsec will be more effective at preventing "traditional" exploit techniques.

But modern software development is so bad, even after using high-level languages and abstractions, much of the webapp and backend development is rife with security holes. You don't need to use things like buffer overflows anymore to simply extract data or take over accounts. Servers are so easily accessible and botnets are so widespread that owning a server isn't really the point anymore; once you have all their data, who needs root?

Agree!

You don't need to bother with old-school stuff like grsec, iptables, IDS, chrooted applications or any stack-protection technologies.

Get a WAF, audit your web-app source-code and use a pen-test tool regularly instead.

SQL-injections walk right in, through the front door. They stuff their pockets full of data and then leave the same way they came, unnoticed most of the time.

Ya, I guess I'm thinking of it more from a hosting/cloud perspective, in that even though one user/tenant's site has been hacked, it would be nice not to have to wipe the machine for all other users.

I am definitely going to reinvestigate putting every user's data in their own container. When I last looked the options were FreeBSD Jails (not appealing) or Solaris Zones (interesting but a whole new OS); now with LXC maybe that can assist? There must be a way to harden things further...

FreeBSD's jails have had a lot of work done on them over the last couple of versions. Now that 10.0 is out, could be worth another look.

Why not Docker? (lxc)

Docker is not designed to isolate the root user from the rest of your system, and containers/capabilities are not MAC-based security. Docker strips 16 capabilities when it runs a container, leaving 23 additional capabilities to be abused by root, and whatever other vulnerabilities are left that capabilities don't cover. To prevent this you need guest isolation via virtualization.

https://wiki.ubuntu.com/LxcSecurity http://www.infoq.com/news/2013/09/docker-container-security http://s3hh.wordpress.com/2013/07/19/creating-and-using-cont...

This is partly because shared hosting providers act like Internet slumlords. Their unwillingness to spend sufficiently to secure their hardware and users' data is not sufficient evidence to conclude that they are secure. (Honestly, at $4 or whatever a month, I sort of understand.)

It is on GoDaddy servers. In my experience, there was a back-door from other 'unsecured' sites.

It depends what you are running: if I have exec then I can drop to root on a lot of systems. But with these kind of scripts its usually an automated process which infects the system and that usually means nothing else was done besides infect all www files. But most likely they could have rooted and thus you need to reinstall.

There's also no point in rebuilding until he thinks the attack vector has been closed. However they got in last time, they can probably do it again.

He was running an old version of Drupal. If you run old versions of Drupal, Wordpress, Joomla etc, you will get hacked.

Running old versions of anything (unless you backport security updates of course, as the Debian project do to keep Stable and OldStable secure without potentially introducing breaking changes with a full package upgrade), particularly popular apps/libs, has this risk. As soon as a project releases a security update there is a good chance that someone out there starts looking for a way to exploit what has been fixed so that they can try use it to abuse sites that have not updated yet.

There's a difference in severity between "will get hacked" and "has this risk". The popular web apps are targeted, and very shortly after any exploits are found, automated drive-by attacks are live across the web. The Googles are great for quickly compiling target lists as these web apps are typically pretty easy to identify - exact phrasing on the login screens etc.

Yep I last year had a locked down aws instance running just a single purpose node.js sever and I saw loads of attempts to access common web based admin pages eg phpmyadmin WordPress.

Naturally these where from Chinese or Russian ip address ranges

"unless you backport security updates of course, as the Debian project do to keep Stable and OldStable secure without potentially introducing breaking changes with a full package upgrade"

And that is why you need to run production systems on large well supported stable distributions, like Debian, and not DudeOS or FunkyNameOS created 18 months ago by two dudes and never updated since.

Definitely. And why if you roll your own packages for any reason (i.e. you need something in a more more up-to-date form than your chosen stable well-supported distribution includes) you simply must keep a close eye on the relevant projects to make sure that you don't miss an important hole filling fix.

FYI: I run Debian/Stable where I have a choice and stick with the provided versions of everything as a general rule, though I currently have nodejs, npm, and some related modules compiled from other sources.

I agree, although it is worth noting that running something obscure will make you less susceptible to automated, untargeted attacks. Potentially quite a lot more vulnerable to anything targeted, though.

I've been building Drupal sites for 6 years, and I've never had a single one get hacked, even after I stopped updating them. It's far more likely that FTP with a weak password was the attack vector.

Not sure about Drupal, but Joomla gets targeted all the time. I don't think FTP is the attack vector. Yes, it's possible the password leaked, but it's more likely a bug in Drupal was exploited.

Disclaimer: I work at a hosting company, and this is my personal experience with hacked websites.

On Joomla I've seen exploits via the site search feature and the admin login (I too work at a company that does hosting). I haven't seen Drupal sites get taken out.

It's also likely that the vector was a vulnerable module he had installed.

Yes I will advise my client to change hosting to a proper hosting company.

I am interested in how it operates. Is it as simple as : "It runs the remote php file and adds whatever html?" or is there more to it?

I find similar functions in all the files (error_404/http_request_custom/getUseragent/getReferer/convertIpToString/getIp)

There is substantially more functionality, most rendered moot by code which allows the attacker to run arbitrary instructions on the server. Typically after observing that (called a "shell" in hacker parlance) one doesn't really spend much time looking at the rest of the code.

What exactly do you mean when you say rebuild from metal?

Anything short of formatting all the drives and reinstalling the OS will not result in a trustable machine. As they said, the attackers most likely have root access and rootkits make the machine lie about it's state. So even if you remove all your website code and replace it, they probably have a backdoor to get in and compromise it again. And you can't find that backdoor, because they modify utilities like 'ls', etc.. so that you can't detect files that were installed or modified by the rootkit.

To be truly sure you're back to normal, you would want to re-flash firmwares, as well. Which is truly a pain. Your NIC has firmware, your Motherboard has firmware, even SATA disks have firmware...

Here's a gentleman who put malicious firmware on a hard disk to bypass linux security by serving a neutered /etc/passwd file. http://spritesmods.com/?art=hddhack

Generally, you have to choose the level of rebuild that you can live with given your likely attacker. Usually, flashing firmware is dangerous and likely to alert the operator to the infection, so most attackers interested in spam/phishing wouldn't try that approach. That is probably some three-letter-organization level stuff.

Yeah, like you hint at--this gets near Dragos levels of paranoia. Firmware based hacks are highly non-trivial; particularly considering the sophistication of the scripted attacks that target Wordpress, Drupal, et al. installations en-masse. Unless it's a very high-value target, most people are not going to go through the trouble of coming up with a custom attack for every bit of hardware they encounter (not to mention the amount of functionality you can squeeze into flash while still maintaining its usefulness as operating firmware). In the amount of time it'd take to do that, you could manually seize tons of other insecure installations.

Also, if you happen to know of any Linux utilities that can flash a live OS's HDD firmware without the system going to shit, I'd be curious to learn more.

That's quite the hack.

Really? Even if the web server was run in a VM?

If this is the case, then EC2 wouldn't work. Anyone with a VM could own the host (since they have root on their own VM?).

No, having root on the VM doesn't necessarily let you own the host (I don't think I implied that), but it's been known to happen: http://en.wikipedia.org/wiki/Virtual_machine_escape. If we're dealing with attacks as sophisticated as others mentioned in the thread, I wouldn't put that outside of the realm of possibility.

edit: at the very least I'd recreate the VM from scratch.