1. Even if you have one way to protect your database (e.g., firewall rules), you should have another. In this case, use a database password or (better) client TLS certificate to authenticate traffic. We're all human and we mess up. You should be designing systems that are graceful in response to your inevitable mistakes.
2. If you can afford another server/a hosting provider with VPC-like features, don't put your databases on the open internet. Run them in a private network (RFC 1918), behind a NAT and a load balancer/entrypoint that only routes to your internet-serving applications. Allow only those application servers to hit your databases. If you had done this, the attacker wouldn't have noticed your mistake, because all they could hit was your public server.
3. Keep regular backups. Oh, and test them! If you don't test your backups, you don't have backups - you have archives. In the GitLab data loss incident [0], they had 3 methods of data backup, but all of them failed. A regular test would have discovered this. Don't make that mistake.
Good on you for sharing the events as they happen. I think people tend to be much more forgiving in response to openness. Don't freak out, and write a public postmortem when you're done.
[0]: https://about.gitlab.com/blog/2017/02/01/gitlab-dot-com-data...
In general, put everything in private subnets, and make the only way any traffic can get to a server is through a load balancer. There are very few reasons to have a server itself have its own public IP address, and using your load balancer as a chokepoint, means you can set up layers and layers of redundancy to prevent traffic from ever being able to reach a database under your control.
This holds true whether we're talking RDS DBs, something you've spun in a K8 cluster, or something you're running on a vanilla compute box.
Assume you will screw up at some point and open up a port that shouldn't be. Ask yourself, "what's the impact here?" If you're OK, even with a fat fingered port, great.
Assume you will have a dev deploy a DB without a password. "What's the impact here?" If you're ok, even without a password, great!
That isn't to say that the above two scenarios are something you should tolerate, but automation can help detect these sorts of issues and make it easy for you to resolve them. While that automation is running, you want to make sure you're not going to get owned.
Kudos to the OP for sharing their story. Always lots to learn on this front, and we can always get better at our cloud operations.
You want to design your system so that if a network a critical misconfiguration occurs you don't open yourself up - you simply stop working.
(Too many years chasing EMR clusters getting dropped onto the internet by users with AWS console access).
1. Be cautious about trendy technologies that promise to make life easy - often they cut corners to do so, and often security is one of those corners
2. Use real authentication rather than network firewalling. Make your datastore TLS-only and require a valid client certificate to connect; that way it doesn't matter if it's exposed to the internet (indeed I'd argue you probably should expose every server to the internet - much like Chaos Monkey, it's counterintuitive but it forces you to build your systems with the right kind of resilience from day one)
As described, you're _only_ relying on client TLS to protect your database. What if the TLS key leaks? So you're down to one layer again - one mistake and it's game over. So maybe you need Hashicorp Vault so you can have client certificates with very short validity periods. So do you expose Vault to the Internet, so you can fetch your client certificate to query the DB? What if Vault has a bug? And it's turtles all the way down ...
I love zero trust designs. But I think saying you can just slap client TLS on the problem and be done is laying the foundation for a repeat of this event - however you look at it, you want redundancy in your security.
If you have 100 devices on a network where everything is unprotected, that’s 100 different ways someone can try to get full access to 99 other devices.
You’re simply pointing out that no amount of layers are foolproof. The goal is not to reduce failure to 0% but to 0.001%.
An easy way to get independent failures is to layer a private network with strong auth and firewall rules. While it's certainly possible to expose your DB to the internet safely - given sufficient protections - you won't get that with just a TLS key. And even if you try to implement the "obvious" additional layers here (Vault, right?), it's easy to inadvertently include design problems that reduce to "only one failure and the system is exposed."
Maybe there is no excuse, but literally every company I've worked for (including Fortune 500s) has been missing at least one item from your list. So "industry best practice" means committing less time and money than it would take to implement all those things properly (rightly or wrongly) and you need to triage and prioritise.
You see this "authN/authZ above all else" line of thinking in Google's security design [0], with their ubiquitous login wall. For their employees, that login wall has extra hardening - you can't do regular password resets, and you need to possess a physical security key (which acts basically as a scaled-down single-purpose HSM) and pass a suite of posture checks on the device (proportional to the sensitivity of the protected resource) to pass through it.
Then they put this login wall in front of everything, even internal services, and that tends to be OK because the system "fails closed," and the only way to access their protected resources is via physical safe rooms deep in the Google offices in which elevated privileges may be obtained.
Putting all your systems on the Internet forces you to get the incentive structure right, and that can be useful in companies where "private networks" can serve as justification to weaken security - if your login page is Internet-exposed, then you simply have no choice but to make it strong enough to withstand the chaos of the Internet. There's significant merit in that.
[0]: https://sre.google/books/building-secure-reliable-systems/
This is in vogue but it’s wrong because it presents this as an either/or.
In reality, people are going to goof up and auth flag at one point, accidentally bind a service, or just run a service with a 0-day (a.k.a everyone).
There is no reason to run a server accepting traffic from every IP if your clients are coming from known ranges.
People see the zero trust model and mistakenly think it means no network-level filtering. This is completely wrong and all of the big players still protect backend services with network ACLs on top of required auth.
I am looking at, make private network, and then you have to explicitly add an gateway (usually a load balancer) that can access it.
In short don't design system where you need to filter, design the system, where you need to take explicite action, to make something public.
This is easily, done on something like AWS.
And his second point is table-stakes as far as I'm concerned. As others have said, do all of them. They are not hard to do.
Have we suddenly stopped teaching the basics?
Everyone seems to want to worry about “nation state actors” and getting with novel 0 days when in fact missing the basic low hanging fruit is likely to result in far more damage
@conesus It's just not worth using discount hosting providers for this exact reason. Use a M(icrosoft) A(mazon) G(oogle) cloud, yes it costs more.... But your time is worth it!
I mean, if you can reliably run Kafka or Postgres in containers, what's sufficiently different about MongoDB?
Or are you talking about using Docker itself as the container host as opposed to K8s or ECS etc.?
There are reasons why running a high performance database instance in containers is problematic, but security is not one of them - not any more so than application containers.
You just need to know what you are doing, it's not a black art.
What's the difference between a container and a instance in a cloud service?
I bet this, in combination with the extremely irresponsible solution to ship mongodb without auth as default has caused countless of data leaks and destruction events. We just haven't heard about most of them. Elastic provides the same foot-gun.
Last year someone deleted almost 4000 open mongodb and elastic databases in what was called the Meow attack [1].
In my opinion it's as irresponsible as HW manufacturers shipping with default passwords, something which finally got the attention of regulators[2]. So I wouldn't be surprised if we at some point see some attempts to keep sw-developers accountable for what they give out.
I have for some time been using the data-oil analogy to describe where we are at. If data is the new oil, and a database is a tanker, then we are at the single hull tanker stage. We need double hulls, but just like the oil industry, the sw industry has little incentive to fix it themselves. I am hoping we get some regulation which improves the situation, because otherwise this will keep happening.
1: https://www.bleepingcomputer.com/news/security/new-meow-atta...
2: https://techcrunch.com/2018/10/05/california-passes-law-that...
Not GP, just hypothesizing. This is what Travis matrixes do, for example.
But it seems that without external pressure (regulatory?), it's not getting fixed at either end.
I shut it down and moved on, we don't use Mongo to date.
You must have compromised the binding to localhost in some way to allow this to happen as MongoDB only listens on localhost by default.
Serious: Listening on localhost-only works in dev environments only. In production, it is not the norm to run the application on the same host as Mongo, especially given what a resource hog Mongo is. So, for practical purposes, listen-on-localhost is actually an obstacle is needs to be disabled first-thing anyway.
You guys know this too, because you do exactly this (and a lot more) on that Atlas thing you guys love to upsell everyone and their grandmother on.
Honestly, it is telling that this is the only defence you could provide — that you listen on localhost — and not anything _actually_ secure in prod. Must come in handy when upselling Atlas, I guess, that your the default configuration conveniently omits everything.
> Did you follow our guidelines? https://docs.mongodb.com/manual/administration/security-chec...
Snark: maybe they couldn't trust your guidelines because MongoDB the company is a known blatant liar[1].
As the first paragraph of my comment says: listen-on-localhost is untenable in production. Unless, of course, you guys seriously believe people should be running their production applications on the same host as mongo daemons. Honestly, I wouldn't be greatly surprised if you guys believe that.
> Our defaults make it difficult to ...
You have one (1) default that does that. Singular. None of your other defaults do that. And, as I've said above, that one (1) default is also useless, because it's one of the first things that need to be disabled in production anyway.
> However if you do add a MongoDB database to a public IP address we strongly encourage you to add a strong password. Better still do not expose your database on the public internet. Put it behind a firewall with auth enabled, secure it with a certificate and only allow access to named IP addresses.
Everybody knows this. You aren't adding anything new. Nobody's claiming MongoDB _cannot_ be secured. Everybody knows that it can be. The question, instead, is: why does every user of MongoDB even need to make it secure?!
I doubt you can answer that honestly, but plenty of us suspect we know it anyway: because MongoDB Inc. "cares" a lot more about developer experience, than it does about their data.
You're awfully arrogant for someone who has no clue in how to properly architect systems.
If you're using Kubernetes then it's very common to have a service mesh in a Production environment to enforce certain safeguards e.g. mutual TLS and provide circuit breaking, auditing, logging etc. In which case MongoDB would be running on localhost.
If you're not using Kubernetes then it's also common to have some form of middleware to achieve the same as above e.g. HAProxy, F5. Again, in which case MongoDB would be running on localhost.
Look, I've done my fair share of fronting services (including DBs) using TLS/SSH proxies and load balancers. When I needed them. But the question isn't if any of that can be done or needs be done. The question is: Why does MongoDB, which has all of these security support built-in, not enable them out of the box?
And your answer to that is ... throw more stuff on top of it?! Are you seriously claiming that every production user of MongoDB should take on so much software surface area just to fix the broken defaults Mongo ships with?! This is worse than what even MongoDB Inc. does; at least they document how to enable security (lol, what a concept) and merely automatically blame the user for all lapses.
And no, k8s service meshes and LBs in front of DBs aren't nearly as common in production as you're claiming.
Everyone has put some sort of middleware between their applications and databases.
Your claim that no one is running databases on localhost is simply your ignorance.
Are you seriously claiming that the only production users of DBs are "Fortune 10, banks, telcos"?! Or are you claiming that because those guys do something a certain way, everyone else must also do it like that?! This is a weird variation of the Argument to Authority, and even more flawed than the original.
> Everyone has put some sort of middleware between their applications and databases.
Really? "Everyone"? Or are you just generalising the state of the entire industry based off of your limited experience with a small number of players in it?
> Your claim that no one is running databases on localhost
I made no such claim. I said it's "not the norm" and that it's "untenable", not that "no one does it". It doesn't matter that there are a few examples you've seen that do; they're not representative of the entire industry.
On the other hand, for the vast majority of the industry who run databases in production, my claims hold.
The vast majority of the industry, that does not include "Fortune 10, banks, telcos".
Docker, as we know will open exposed container ports to the world, that shouldn't really be the baseline though for not having your instance compromised in less time than it takes to enter an iptables rule correctly, or read the guidelines.
I'm not trying to place blame, it was an exploratory endeavour anyway, but Meow existed because security in Mongo is a guideline and not a rule
As someone who builds secure software solutions for a living it doesn't thrill me that security is often an "optional extra" (looking at you elastic).
If I asked our customers/users the same question you just asked me, and then followed it up with "You must have compromised...", I'd be in hot water.
A combination of factors contributed to us choosing not to use Mongo at the time, if we have such a need again, it will be considered without prejudice.
The first time I commented in this whole discussion was _in reply to_ your "you must have compromised ...".
Only if you choose to explicitly expose them.
In which case the fault is entirely with you.
Wow, the level of entitlement around open source is appalling. People download and run shit for free and get mad that they didn’t configure it right and want the government to go after the maintainers.
I can’t wait for the day where us open source devs have to contribute patches via pseudonyms and tor because they aren’t “government compliant”.
If some philanthropist were to give out free bicycles to everyone, but it turns out that unless you tighten a bolt one of the main support bars will likely snap and could even impale you, the government would rightfully go after them regardless of any "without warranty" disclaimer or EULA.
This is already well-trodden legal ground in the physical space; it's just a matter of bringing it into the virtual space.
Open source projects seldom get popular overnight. Responsibilities grow as you get more popular, that is all...
That is the GP example. If I make 1 bike by hand and put that somewhere on the street, and someone takes it and breaks their legs, that is not my fault. Nor should it be. If I sell that 1 bike, that can be another story.
It will kill open source imho as if you, in your underwear in the attic are responsible for some code you write and put on GitHub, you won't write that code anymore because of the risk; but I don't want to find out who is right here.
Responsibilities should befall the person deploying the software. Aka if you write some software and throw in on GitHub, that should not do anything. But if I take that software and put it in my app or backend and takes people's money because of some bug, then I am the one who is responsible. And this is already the case anyway. No changes are needed.
I think we agree though as I think we are talking about the person who writes and deploys/sells the software: they should indeed be responsible and already are.
Except with GDPR, which makes you responsible for having regular security testing, I mean, just an nmap after changing your infra, it takes 2 minutes, and yet when you don't take these 2 minutes you find a way to blame the manufacturer. I mean, don't you /know/ that bots are scanning the internet for open unauthenticated services, and still want to be taken seriously. Back in the days every kid was doing it to host warez you know.
I am also making the observation that this does not seem to get fixed by itself, rather it seems it needs regulation, which includes GDPR.
I've commented here in the past about my feelings on Docker's attitude towards real issues raised by users, the only way I can describe it succinctly is "contempt".
There's some serious flaws, with tickets that are the best part a decade old now, one simply can't rely on them to do anything.
Stop using Docker or solve the issue yourself is my advice.
The userspace proxy used by Docker Swarm is still trashing my L3 client IPs though, yet another docker networking hassle.
Ultimately solely relying on an L3 firewall for access control is a bad idea, though. Your services should all have authentication turned on, even if they are only bound to localhost, for reasons which at this moment must be obvious.
The root cause is that IP packets going to the containers are not going through the INPUT chain of the "filter" table (they go through FORWARD), while various firewall projects like ufw or firewalld only provide convenient management of the INPUT chain, and, worse, don't event provide a good way to express the notion of "packets going to external port 27017 and then forwarded" (i.e. the equivalent of the --ctorigdst option provided by raw iptables).
Here are some options for container engines:
a) Use only a userspace proxy (which is what Docker does when configured with {"iptables": false}). This way, there is no packet forwarding, so packets go through the INPUT chain, just as expected by high-level firewall packages like ufw or firewalld. The major downside (which makes this method useless e.g. for containerized mail servers) is that the information about the source IP is lost, and there is no good way to fix this. I guess TPROXY can help here, but nobody uses it.
b) Use slirp4netns (which is what Podman does when running rootless). It has a really nice mode (available via "podman run --network=slirp4netns:port_handler=slirp4netns") where on the host side, there is only a userspace process listening (so that packets go through the INPUT chain), but inside the container, packets going out of tap0 have the correct source IP. The downside (actually a Podman limitation) is that you can't set up multiple containers communicating over internal IPs.
I would say that I am not really in favor of options (a) and (b) because of the overhead created by the proxy or by slirp4netns. If port forwarding can be done in the kernel (and it can, the only missing piece is --ctorigdst in high-level firewalls), it should be done in the kernel.
c) Document the situation better (e.g. I don't see the --ctorigdst option mentioned at all on https://docs.docker.com/network/iptables/), shift the blame to firewall authors so that they start creating a duplicate of each INPUT rule also in the FORWARD chain with --ctorigdst added as necessary.
d) Provide usable primitives (similar to Network Policies in Kubernetes) to control the firewall for containers, so that ufw or firewalld is not needed.
*** WARNING YOUR FIREWALL ISN'T WORKING!! RUN AGAIN WITH --my_firewall_is_broken_and_I_accept_the_risks OPTION TO CONTINUE!!! SEE THIS FOR MORE INFO: <hyperlink to docs> ***
* Install slirp4netns
* DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER="slirp4netns"
* Upgrade to rootless docker 20.10.7 or higher (was bugged in 20.10.6. Yup.. bad.)
We have a strict policy that servers must not have a routable IP address at all, and must have an external firewall applied. It's a good idea in any case, but turns out it's absolutely necessary with Docker.
Anyone else running docker and using iptables really needs to read this https://docs.docker.com/network/iptables/
'The Redis security model is: “it’s totally insecure to let untrusted clients access the system, please protect it from the outside world yourself”.' -- http://antirez.com/news/96
That blog post also helpfully shows how to write you own key into .ssh/authorized_keys to you can log in as the redis user over ssh. From there use your favourite lunar priv escalation bug to p0wn the box completely. (Or just run your cryptominer as the redis user...)
Note: that's about 5 years old now...
https://github.com/redis/redis/commit/edd4d555df57dc84265fdf...
Eventually a kind script set a password on redis which caused me to notice and fix this issue.
I am not sure anymore how i solved it but after some time (and none of the documented solutions working) i decided to let docker do to iptables whatever it wants to and used another firewall which filtered the traffic again after iptables was done "filtering" it.
And these docs read a lot like: “We are going to totally ignore any of your firewall rules unless you follow all these steps exactly and do a lot of manual work.”
Other than that, it's commendable that you have working backups and are responding calmly and with a plan. I hope you get everything back in working order smoothly :)
[0] https://www.theverge.com/2016/3/8/11179926/facebook-account-...
the case you mention is more of a hacker feat because that exploit had to be crafted specifically for facebook. meanwhile in this case it was most probably someone who just continuously scans the IPv4 address space for open mongo instances and applies the same generic "exploit" against them if it finds some in a fully automatized process
The terms get clouded a bit - but in my definition a "script kiddie" is pretty much this: someone with not much skill(like a kid), but hands on some hacker tools/scripts - to find easy targets and feel powerful. And later on, try to make some money.
And they can make great effort in doing so - but they remain script kiddies. They don't really know how to hack.
Whether this was just a "script kiddie", I doubt. More a professional ransomware gang. But what op probably meant was, it was not a targeted attack.
And DO still have this in motd for newly created droplets:
Welcome to DigitalOcean's 1-Click Docker Droplet.
To keep this Droplet secure, the UFW firewall is enabled.
All ports are BLOCKED except 22 (SSH), 2375 (Docker) and 2376 (Docker).
Though it links to https://do.co/3j6j3po and it mentions ufw problem:
> Note: The default firewall for the Docker One-Click is UFW, which is a front end to iptables. However, Docker modifies iptables directly to set up communication to and from containers. This means that UFW won’t give you a full picture of the firewall settings. You can override this behavior in Docker by adding --iptables=false to the Docker daemon.
This is just what another poster commented on, sacrificing security for ease of use.
DigitalOcean Support Thursday, March 15, 2018 9:53 PM
Hello,
Thank you very much for bringing this to our attention.
I will create an internal escalation to our images team to review this. :)
[...]I'm always using this with providers that support it, since I've managed to mistakenly open ports that should be private (either by misconfigured firewall, or due to the Docker "issue").
[1] https://docs.digitalocean.com/products/networking/firewalls/
Disappointing response, this.
What data got leaked? Please let haveibeenpwned.com know if your system leaked emails or worse.
Actually it makes it worse, since your security is so bad that any "script kiddie" can hack into your system.
If your mongoDB server is exposed to the Internet it will show up there. When that happens, it's only a matter of time until someone targets you.
You can write an alert that probes for sensitive services exposed to the Internet. In that way, if this happens again, you get an alert that you can use to detect the problem early.
Also... use authentication for your database, it doesn't take much effort to do. With a secure password.
If you have a membership (which is a one-time payment of $49 - no subscription necessary) then you can monitor up to 16 IPs. We have a lot of individuals that have configured monitoring for their home network just in case something accidentally gets exposed.
Are they scanning networks 24/7
I’m just a noob in security so therefore learning
https://help.shodan.io/the-basics/what-is-shodan
The scanning algorithm is mostly just this:
1. Generate a random IPv4 address
2. Select a random port from a list of ~2k ports
3. Check the random IP on the random port
4. Store the result of the check
5. GOTO 1
The above loop runs endlessly and because IPv4 is fairly small it doesn't take long to check everything.
Abstractly and intuitively, IPv6's massiveness would seem to put an end to the interesting closed loop of address space vs backhaul capacity that has developed around v4. I can't help but wonder though - with for example some providers leasing out ginormous blocks of address space according to fairly predictable patterns (and customers just using the first v6 address that pops out - if at all), this makes me wonder if it'll be possible to steer v6 scans using a mix of statistics, machine learning, and Perl if statements :).
The other thing I'm idly curious about is how you actually scan on a regular basis. Broadly speaking about long-term viability, I guess the TL;DR probably boils down to coordination and careful nurturing of reputation similar to what the large-scale email providers maintain. But from a technical perspective, I do wonder if/how much things like peering, and BGP, and noise-cancelling routing (if you will), etc, come into the picture - and how big the links are :D
I would be very happy to coincidentally discover writeups touching on these questions anytime. Thanks for reading :)
But for now we can do this with the v4 parts that are left: https://www.youtube.com/watch?v=nX9JXI4l3-E
Also, some crazy person good-haxed a bunch of routers and modems back in 2012 and made http://census2012.sourceforge.net/paper.html (without access to fast connections and before the advent of masscan and other straightforward tools, too). Of note is that the "Unallocated" grey areas in the the analysis images are a curious illustration of how much less-full the IPv4 internet was just ~9 years ago.
nbset:PRIMARY> show dbs
READ__ME_TO_RECOVER_YOUR_DATA 0.000GB
admin 0.000GB
local 16.471GB
newsblur 0.718GB
nbset:PRIMARY> use READ__ME_TO_RECOVER_YOUR_DATA
switched to db READ__ME_TO_RECOVER_YOUR_DATA
nbset:PRIMARY> show collections
README
system.profile
nbset:PRIMARY> db.README.find()
{ "_id" : ObjectId("60d3e112ac48d82047aab95d"), "content" : "All your data is a backed up. You must pay 0.03 BTC to XXXXXXFTHISGUYXXXXXXX 48 hours for recover it. After 48 hours expiration we will leaked and exposed all your data. In case of refusal to pay, we will contact the General Data Protection Regulation, GDPR and notify them that you store user data in an open form and is not safe. Under the rules of the law, you face a heavy fine or arrest and your base dump will be dropped from our server! You can buy bitcoin here, does not take much time to buy https://localbitcoins.com or https://buy.moonpay.io/ After paying write to me in the mail with your DB IP: [email protected] and you will receive a link to download your database dump." }Heavy fine yes but not arrest AFAIK. Anyway this is a script programed to scary the target.
Do you even store personal data inside that database?
Newsblur is an American org. GDPR is a foreign law that has no relevance to American firms lol.
<insert Saruman "you have no power here" meme>
I couldn't agree more with the spirit of your comment, but sadly the reality may be somewhat more nuanced:
GDPR in the USA https://www.cookiebot.com/en/gdpr-usa/
"The GDPR has extra-territorial scope, which means that websites outside of the EU that process data of people inside the EU are obligated to comply with the GDPR. ... In fact, the very first GDPR enforcement was against a Canadian company... being a website in the US does not exempt you from GPDR compliance and the territorial distance will not protect you from its enforcement either."
Reminded me of:
CISA amendment would allow US to jail foreigners for crimes committed abroad https://www.theguardian.com/technology/2015/oct/22/cybersecu...
https://www.axios.com/china-hong-kong-law-global-activism-ff...
In reality, a US business with no EU presence only has to follow US laws. The only "enforcement" power the EU has would be to order the website to be blocked in the EU, and I'm pretty sure they can't even do that.
The data is already leaked, let your users know what was leaked and recover from there.
See also: 80% of orgs that paid the ransom were hit again https://news.ycombinator.com/item?id=27552611
I paid Samuel and entrusted him with my data. Not too much, but enough for it to matter. When faced with a massive leak like this, he downplays everything, calls the hacker a "script kiddie" and calls this "good practice for what will be the first of many sleepless nights", looking at it only from a "service disruption" perspective.
So far we've gotten no indication of what's been leaked, if it contains deleted feeds, or what he's doing to prevent the data from being leaked by the hacker, if anything. He's been solely focused on restoring the service and ignoring the leak. Compared to not having access to an RSS reader for any random period of time, the leak is orders of magnitude more serious to me and I'd wager to most of Newsblur customers.
I honestly don't care if paying a ransom or interacting with the hacker makes him more likely to be targeted in the future, his duty towards his customers was to keep their private data private and not only he failed at that, but he doesn't even seem to register that as his main priority. As far as I'm aware, if he allows the data to leak publicly, then there's no "recovering from there", he's not getting any more of my money.
I'm saying whoever is ransoming the data already has the data, the data is out of Newsblur's control, therefore the data is already leaked.
The data leak is past tense. It has already happened, not will happen. No amount of money will undo that. If that means they've lost you as a customer, that's how it is.
What we now need to know is what data was leaked?
[1]: which to be fair Newsblur, they are, but if a script kiddie hacks you using something so basic as a missing firewall rule.. Arguably not knowing Docker's quirks but using it anyway is the same damn thing as what script kiddies do. Sys kiddie if you will.
[2]: Why is that cause for celebration? Do you not have backups?
The attacker has offered to not publish if they are paid. Their word probably isn't worth much, but $1,000 seems like an affordable sum for a business to gamble on them being honest about it. And if Newsblur doesn't fix their security problems they'll be targeted again either way.
As someone who has a decade of data in Newsblur, if there's any chance that an affordable ransom will keep my data from spreading further I want Samuel to take it.
https://www.shodan.io/search?query=READ__ME_TO_RECOVER_YOUR_...
Really? If they ask 100K dollars they are probably not going to be paid. So, just hijack 100 servers and assure that they will pay (since 1K dollars is "not" much if you are running a business).
Have you contacted the relevant authorities?
Your adversary was unsophisticated but this incident was your fault.
You can then expose ports to a specific IP and use UFW to allow it.
Much cleaner than any UFW-docker hacks out there, and more secure.
Do you know if they simply encrypted the data in place or if they succeeded in exfiltrating a full copy?
Seems like it would be a dead simple detection system, and would be a huge value-add.
You could of course order an additional IP and do it yourself, too, but it seems like the colo doing it would be more efficient in terms of scale. (And they'd likely be more diligent in doing it, avoiding obvious DoS vectors like triggering the honeypot from AWS netblocks.)
Hetzner (my host) sends me nastygrams from their IDS when my box tries to connect to RFC1819 space (which isn't even routable via them!) when running p2p software like ipfs. You'd think if they are willing to complain to customers about zero-impact stuff like that, they'd be willing to blackhole non-customers for nonzero impact malicious traffic.
Not saying it's a good practice but it's a common pattern I've seen.
Yeah I had the same problem. If you are not using orchestration engines like Swarm of Kubernetes you can just avoid it by explicitly binding the port to your local interface (i.e. -p 127.0.0.1:27017:27017 instead of just -p 27017:27017)
I’ve been got by this too.
It’s in the name, so it doesn’t seem worth disputing that at least at the time when the term was coined- it did have to do with age
> or even skill
Skill is literally the defining feature.
What exactly do you mean by “it’s simply a hacker term” anyway? Are words just sounds we make with our mouths?
1. you put your DB in a server which is exposed to the internet.
2. you have no VIP/NAT in front of your systems.
3. you rely in iptables , while knowing some automatic system is manipulating it.
3 hours? I wonder it took so long. I expect this infrastructure will be a script kiddies party room within a few minutes.
1. I need to be able to connect to my DB from anywhere.
2. No idea what that even means.
3. Don't know. Never even touched the firewall.
I have a PW on my DB and that's it. Why do I need more than that?
- put the database in a virtual private cloud (VPC), an internal network
- setup a Virtual Private Network (VPN) also placed in the same VPC from which developers can connect to to access the internal network
- setup at least two MongoDB users, one `readWrite` user that can connect from the internal network and one administrative user that can only connect from localhost
- setup a key based SSH connection only accessible from the VPN to the MongoDB instance
- setup Security Groups (firewall) to lock all the unused ports and IP origins out
That way you'll need a VPN key, an SSH key and the MongoDB admin user's access to fully compromise the database.
Understandable and reasonable. there are ways to achieve this without exposing the DB but it take some more effort.
> 3. Don't know. Never even touched the firewall.
Firewalls are good, because they add an extra layer on security. Personally I believe, there should be firewall on most servers (firewall should be the default). But firewalls aren't magic and they are just one layer.
> I have a PW on my DB and that's it. Why do I need more than that?
Because this means you have 0 margins for error. Either you PW auth works flawlessly and without bugs, or you are screwed.
Also you expose one more endpoint that can potentially be a ddos target.
So do I. So I setup all my dbs with TLS mutual auth or equivalent. Even databases that don't support TLS natively (e.g., Redis 5 and below) get a TLS/SSH port-forward setup for them at the network boundary.
> I have a PW on my DB and that's it. Why do I need more than that?
If you're not using an MITM-proof connection (e.g., TLS or SSH), and you connect to your DB from a network that has me (maybe we're in the same coffee shop, maybe I'm working in your office, or maybe I just work at an ISP between you and your server), then I have your PW.
Had some fun using mongodb but I don't think I'll ever use it again =/
I setup alerting rules so that certain subnets or servers suddenly being exposed to certain instances of blackbox will wake people up!
As others commented, scanning the whole Internet is even not a problem so scanning a "limited" part where you are likely to see these services pop up is even less of a problem.
I think the takeaway is that you cannot hide in the masses on the Internet anymore, 10-20 years ago you could throw up a insecure server and it could be fine for a long time.
Nowadays you must assume someone will find and try to login to your service, even if you put it on a non-standard port.
Also, if it's a HTTPS service take note they when you get a certificate you will be announcing that domain to the whole world and publish it to a searchable database (for example https://crt.sh/ ).
better still use MongoDB Atlas and get our best security practices baked in.
With many people doing this, it is kind of surprising that it took so long for a vulnerable server to be discovered.
[0] https://zmap.io/
* Ansible
* Docker
* Big redesign
* New database cluster
* New firewall config
I have found great benefit to breaking problems down into smaller parts even though some times it causes some extra work.
I appreciate your transparent account of the situation but what does it say about your company if your database got popped by a “script kiddie”?
At least they’re being honest up front. but yes it does show a naivety and inexperience of basic security.
Computer security is hard enough without loaded footguns like these lying around.
Defaults are often insecure but maximise interoperability or general usability. Look at Windows !
The defaults should be secure with explicit unlock steps for those that know their environment well enough that they can explicitly relax some restrictions.
Such drastic changes to the security model should only be one after explicit instruction.
The number of companies that operate without access controls between servers on the same segment is unfortunately quite large, database security controls are - again - more often than not left at their default setting and those too are quite often insecure.
Defense in depth always has limited depth, though I totally agree that running a database without access controls is not the way to go.
The main problem is lack of basic security.
I'm now self-hosting a rss reader, but NewsBlur will always remain dear to my heart.
[0] https://github.com/docker-library/redis/issues/259#issuecomm...
Oh boi. I just knew(0) that default value is problematic.
Too bad this time it caused somebody real money.
Angry at myself for not reading the docs carefully but who has time for that? :/
Is this a useful distinction? Define the sharp line between a "script kiddie" and a "hacker".
You're describing a person who detected your vulnerability and acted within a 3 hour window of it appearing and closing. If I were you I'd be more focused on reflection and not on putting labels on whoever hacked you.
This is crazy. Your network should have been on a private IP address space behind a firewall running static NAT exposing only ports 80 and 443 on a routable IP address. This is network architecture 101.
https://www.imperva.com/blog/ransomware-attacks-on-mysql-and... https://www.itproportal.com/news/ransomware-attacks-on-mongo... https://security.stackexchange.com/questions/237048/mongo-db...
Everybody falls for that, I mean, look at the BTC these guys made, it's crazy! Anyway, Docker uses the DOCKER-USER firewall chain:
https://docs.docker.com/network/iptables/
Example:
https://yourlabs.io/oss/yourlabs.docker/-/blob/master/tasks/...
People should really test their firewalls after setting it up.
Another thing, instead of using Ansible+Docker and exposing ports like that, use Ansible+Docker-Compose, so that your containers of a stack have their own private shared network, then you won't have to publish ports to make your services communicate.
https://docs.ansible.com/ansible/latest/collections/communit...
This situation is more of a script kiddie than a hacker. I'm in the process of moving everything on NewsBlur over to Docker containers in prep for the big redesign launching next week. It's been a great year of maintenance and I've enjoyed the fruits of Ansible + Docker for NewsBlur's 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and soon ML models).
About two hours before this happened, I switched the MongoDB cluster over to the new servers. When I did that, I shut down the original primary in order to delete it in a few days when all was well. (Thank goodness I did that! It'll come in handy a few hours from now).
Turns out the ufw firewall I enabled and diligently kept on a strict allowlist with only my internal servers didn't work on a new server because of Docker. When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world. So while my firewall was "active", doing a `sudo iptables -L | grep 27017` showed that MongoDB was open the world. More info on SO[1].
To be honest, I'm a bit surprised it took over 3 hours from when I flipped the switch to when a script kiddie dropped NewsBlur's MongoDB collections, and ransomed about 250GB of data. I am now running a snapshot on that old primary, just in case it reconnects to a network and deletes everything. Once done, I'll boot it up, secondary it out, and be back in business. Let's hope my assumptions hold.
[1]: https://stackoverflow.com/questions/30383845/what-is-the-bes...