Edit: Im aware its not truly immutable (read only) but you can reset your environment very easy and patching also becomes easier.
Containers have essentially 3 advantages:
- Restart the containers after they got pwned, takes less than a second to get your business up and running again.
- Separation of concerns: database, reverse proxy, and web service run in separate containers to spread the risk, meaning that an attacker now has to successfully exploit X of the containers to have the same kind of capabilities.
- Updates in containers are much easier to deploy than on host systems (or VPSes).
And, either way, do you have a thought on whether you'd still prefer a docker approach?
I have some on-prem "private cloud"-style severs with proxmox, and just curious about thinking through this advice.
Sorta: yes the container is immutable and can be restarted, but when it does, it has the same privs and creds to phone up the same DB again or mount the same filesystem again. I'd argue touching the data is always the problem you're concerned about. If you can get an exec in that container you can own its data.
For VMs, they did take off and essentially the entire cloud ecosystem runs on mostly VMs behind the scenes for VPS and similar hosting.
It's true though at it seems more popular for developers to reach for containers when they need to think about deployments, particularly docker containers. But VMs are still widely in use and deployed today.
(I helped build digitalocean from zero the pre-IPO, so I'm verrry rusty, this all might be nonsense/wrong think, and happy to be told as much! :))
On SSH password auth: its secure if you use a long, random, not reused elsewhere password for every user. But it is also very easy to not do these things. SSH certs are just more convenient imo.
For ssh, the problem does not lie within password auth itself, but with weak passwords. A good password is more secure than a keypair on a machine whose files you can't keep private.
If ssh password out would be no problem, then why are so many APTs "wasting" their botnets with credentials stuffing?
Your assumption is wrong, and policies for key based auth eliminate the problem quite easily. Versus on the other hand: are you checking every colleague's password for length, charset, etc? All the time? On every server?
Probably not.
- The Docker daemon runs as root: any user in the docker group effectively also has sudo (--privileged)
- Ports exposed by Docker punch through the firewall
- In general, you can break the security boundary towards root (not your user!) by mounting the wrong things, setting the wrong flags etc.
What Docker primarily gives you is a stupid (good!) solution for having a reproducible, re-settable environment. But containers (read: magic isolated box) are not really a good tool to reason about security in Linux imo.
If you are a beginner, instead make sure you don't run services as the sudo-capable/root user as a first step. Then, I would recommend you look into Systemd services: you can configure all the Linux sandboxing features Docker uses and more. This composes well with Podman, which gives you a reproducible environment (drop-in replacement for Docker) but contained to an unprivileged user.
It's unfortunately very common to install, for example, a project as the "ubuntu" user and also run it as the "ubuntu" user. But this arrangement effectively turns any kind of file-overwrite vulnerability into a remote-execution vulnerability.
Owning executables as root:root, perms 0755, and running as a separate unprivileged user, is a standard approach.
To take this further, it's also possible to enforce that executables are confined to a few well-understood directories with good permissions by using the noexec option when mounting. Directories with executables are mounted normally (with locked down write permissions) and everything else mounted with noexec.
I use this on Nix OS. There all executables live under the read-only directory /nix/store. So you can mount / with noexec, giving the nice property that all files that are writable are by definition not executable.
Docker is for organizing things for yourself, just like directories are. If you want actual isolation you have to take extra steps.
Services have the following dependencies: static data files; configuration files; executable code/binaries; library dependencies.
In days of yonder, you'd need to download/install all of that ^ on each machine where "service A" needs to run. Developers would run and test "service A" on ubuntu 18.04. But production servers had to run ubuntu 16.04 because "service X" that also runs on the same server needs a library that has not been ported to 18.04 yet.
But "service A" needs a library that was never available on 16.04. Welcome to dependency hell!
Containers bundle all of those dependencies into one object that can be downloaded directly onto the host server, ready for the "service A" process to execute. Now it doesn't matter if production servers are running 16.04. Everything "service A" needs is stored inside the container blob (including some minimal ubuntu 18.04 stuff).
the magic that lets this happen -- containers re-use the host server's OS kernel. Running a new ubuntu 18.04 container does not start a new OS kernel running. the process for your container is just 'firewalled' off from all other processes using cgroups [0]. containers re-use the host's kernel, start a cgroup'd process which starts your container's services and processes (the 18.04 'OS' services and your binary/code/executable).
short/simpler version: containers share the core of the underlying operating system on the host server.
> If you want actual isolation you have to take extra steps.
unfortunately, this means containers share the core of the underlying operating system on the host server.
containers not being isolated from the host server OS can present a security risk as you can escape from the container and "do bad things to host server". [1]
In cases where that is a problem you mostly have two choices:
* use VMs instead (a completely isolated OS instance is started for each service, cannot interact with the host OS at all -- this uses a lot more memory/cpu)
* use rootless containers [2] (container processes are launched under a specific user namespace rather than kernel namespace -- escaping the container means you only get access to the user namespace)
[0]: https://en.wikipedia.org/wiki/Cgroups
[1]: by default the docker daemon service and all the container processes it starts are running as root, which means escaping out of a container in a a default docker installation is as bad as giving someone root.
Docker containers don't have mount permissions by default.
I've been using ufw-docker [1] to force ufw and docker to cooperate. Without it, Docker ports do actually get exposed to to the Internet. As far as I can tell, it does its job correctly. Is there another problem I am not aware of?
The idea is to make your app immutable and store all state in the DB. Then, with every deployment, you throw away the VM running the old version of your app and replace it with a new VM running the new version. If the VM running the old app somehow got compromised, the new VM will (hopefully) not be compromised anymore. In this regard, this approach is less vulnerable than just reusing the old VM.
It might look something like:
1) get funds via illegal activity (dirty funds) 2) spends funds at an ecommerce site (dirty funds) 3) secure a paypal refund WHICH GOES TO ANOTHER ACCOUNT (clean funds)
The PayPal vulnerability allows the money to move from a dirty chain to a clean one.
How it breaks the chain?
Account1 buys for 10k USD, requests refund, receives it?
Even if it went for some reason to account2 then there is still the chain, but why would it go to other?
A money launderer can use a marketplace by creating a seller account and buying from himself. Since he’s the one buying he doesn’t need to deliver anything but he gets the money from a legit source. Usually he would use a payment method as close to money as possible so that it leaves less traces. But in OPs case, the amounts are low so he needs too many transactions to get something valuable. And because of the disputes, he’s (probably) not getting the money (?).
It could be card testing: the fraudster has a bunch of cards and doesn’t know which is valid or canceled. The best way to find out is to test in a real site. So he’ll test out each of them and the ones that go through are good to use elsewhere. The thing is that it would be better for him not to dispute the transactions so the OP would take much longer to find out about the scheme and shut it down. It’s better to use low amount transactions in this case so it doesn’t use too much of the credit available for him to defraud and probably doesn’t warn the card owner.
Another option is doing it just to hurt the OP marketplace. If you have too many disputes the brands can fine you and if you don’t solve the problem they can turn your account off. I’ve seen it happen when a competitor was trying to hurt the e-commerce. It’s a low move and rare but it happens.
One thing that might help is to analyze the sellers too. In a money laundering and even in the other settings, it could be part of the scheme. Are they new accounts? Are their volume exploding out of nowhere? Etc
This only works (in my mental model), when you produce the product you're selling in-house – like a digital product. But lots of "reselling" type businesses try to use this scheme as well. Like a restaurant might ring up more meals than they served, or less to not pay taxes. But, is this not easily spotted when the food import(?) cost doesn't match the revenue?
Maybe I just answered my own question, if the business is able to cook the books both ways, but it would also limit how much they're able to launder. Or is the import/export balance rarely/never checked?
More likly the above are selling something illegal though. Pay for the expensive hand car wash but get drugs instead with a cheap automatic wash - nobody will know the difference.
For higher valued goods they use horses. A saddle can go for $30,000, so you buy some $1000 saddles and sell them for $30,000 and $29,000 worth of something else.
Especially if you stick with quarters instead of using game cards like most modern arcades. Since quarters would be recycled anyways (Taken from the games and restocked into the quarter machine), it makes it easy to just deposit the cash you want to launder as if it had been fed into the quarter machine.
The reason for this is stealthy botnets.
For a brief rundown, I'd suggest this article.
https://jan.wildeboer.net/2025/04/Web-is-Broken-Botnet-Part-...
I am aware of these types of botnets, how they work, and which companies are behind them. Hence the reason for adding my spam database to the initial comment, which focuses on exactly those, combined with the ebpf firewall module that analyzes and correlates repeated bad behaviors.
It's not a new technique btw, APT28/29 and others have been doing this for around 10 years now.
It's also a bug in the paypal API that they're abusing, where the SDK doesn't differ between example.com and www.example.com. If webshops like yours get exploited and used for money laundering, they will mix transactions from those two subdomains, while leaving the www.example.com domain as it is. The support people at paypal are dumb enough to not take care about each case, and usually they mix transactions later also via other social media services that have microtransactions (e.g. tiktok or snapchat streams where you can gift away items).
The way paypal support's workflow works is that they have to nanually identify each and every transaction separately, meaning a human will be busy for weeks on end. Not kidding you. That's how the scammers keep winning with schemes like this. Usually there's also no way to escalate this, not even for business customers, at paypal, due to how their support offices are structured organizationally.
As a mitigation I'd recommend to block ASNs that are known hosters that do this, and double check your webshop version for known vulnerabilities and fixes.
If you don't use docker already, start to virtualize your webshop software now. I can't stress how important this is. Also double check any users and passwords you are using for the services.
If you need a starting point to block those botnet affiliated networks, I started both a firewall and scam database project that does exactly this:
[1] https://github.com/cookiengineer/antispam
[2] https://github.com/tholian-network/firewall