Anything short of formatting all the drives and reinstalling the OS will not result in a trustable machine. As they said, the attackers most likely have root access and rootkits make the machine lie about it's state. So even if you remove all your website code and replace it, they probably have a backdoor to get in and compromise it again. And you can't find that backdoor, because they modify utilities like 'ls', etc.. so that you can't detect files that were installed or modified by the rootkit.

To be truly sure you're back to normal, you would want to re-flash firmwares, as well. Which is truly a pain. Your NIC has firmware, your Motherboard has firmware, even SATA disks have firmware...

Here's a gentleman who put malicious firmware on a hard disk to bypass linux security by serving a neutered /etc/passwd file. http://spritesmods.com/?art=hddhack

Generally, you have to choose the level of rebuild that you can live with given your likely attacker. Usually, flashing firmware is dangerous and likely to alert the operator to the infection, so most attackers interested in spam/phishing wouldn't try that approach. That is probably some three-letter-organization level stuff.

Yeah, like you hint at--this gets near Dragos levels of paranoia. Firmware based hacks are highly non-trivial; particularly considering the sophistication of the scripted attacks that target Wordpress, Drupal, et al. installations en-masse. Unless it's a very high-value target, most people are not going to go through the trouble of coming up with a custom attack for every bit of hardware they encounter (not to mention the amount of functionality you can squeeze into flash while still maintaining its usefulness as operating firmware). In the amount of time it'd take to do that, you could manually seize tons of other insecure installations.

Also, if you happen to know of any Linux utilities that can flash a live OS's HDD firmware without the system going to shit, I'd be curious to learn more.

That's quite the hack.

Really? Even if the web server was run in a VM?

If this is the case, then EC2 wouldn't work. Anyone with a VM could own the host (since they have root on their own VM?).

No, having root on the VM doesn't necessarily let you own the host (I don't think I implied that), but it's been known to happen: http://en.wikipedia.org/wiki/Virtual_machine_escape. If we're dealing with attacks as sophisticated as others mentioned in the thread, I wouldn't put that outside of the realm of possibility.

edit: at the very least I'd recreate the VM from scratch.