I think that's fair, and I had the same initial reaction. However, people are generally fine with DMARC or other crowdsourced spam signals, so it's not clearly bad.

I haven't looked at the source, but it should be easy to turn the crowdfiltering off or self-host if you have your own larger enterprise, no ?

It also depends on what exactly is transmitted. Transmitting all IP addresses in the clear might be bad, sending out hashes might be better ?

I think it's actually a good idea to try to detect those IP at scale. On your own individual server, you just see a million login attempts from a million different nodes on a botnet. But if you have millions of servers report the bad IPs and aggregate, you can see patterns and hopefully catch whole botnets.

> However, people are generally fine with DMARC or other crowdsourced spam signals, so it's not clearly bad.

DMARC isn't a crowdsourced spam signal. And from my memories chatting with people running mail servers—the crowdsourced spam filtering is fairly controversial.

Apologies, I was thinking of the spamhaus list, which I'm also not sure now is crowdsourced in the sense most people would use.

But most email providers build spamfilters using 'crowdsourced' techniques between their users (if enough users mark a sender spam it might be marked spam for all users).

And while that may be controversial, it's also highly effective.

Speaking from personal experience working with these reputation warehouses and with handling a good deal of incoming and outgoing email volume, they aren't any more effective than other IP-based spam heuristics and if everyone configured their DKIM correctly these warehouses would be largely obsolete. I don't see a problem with crowdsourcing heuristic data per se, but email is a much narrower field than internet traffic at large and I can see a lot of the ways that poorly configured fail2ban type software could accidentally ban a lot of valid traffic.

> Transmitting all IP addresses in the clear might be bad, sending out hashes might be better

How many IP adresses are valid? Can't you hash them all and compare them with the hash you receive?

2^32 IPs is 4GB * 4 Bytes (1 IP) + 4GB * size of the hash should be the needed space I think.

1) That is not what DMARC is.

2) I won't be using any "security" tool that automatically shares details on active mitigation efforts.

> I think it's actually a good idea to try to detect those IP at scale.

It is! But not with my machines.

Sorry, we should have made it clear, it is totally optional. You just don't get the IP rep DB part of the soft if you don't share, but the behavior is still 100% functional.

This sounds terrible. I’ve locked myself out of a machine once because I remembered the password wrong. That was annoying. I might have done it without noticing when trying to log into the wrong machine accidentally more than one. Locking myself out of all machines would be way worse.

I mean you could also imagine someone somehow seeding your ip address into the crowdsourced fail2ban to maliciously deny you access to your machine for maintenance, or to trigger some sort of opsec regression to get access.

Spoofing an IP is not exactly hard, so ... yeah, I'm a bit skeptical of the idea.

Yet at the same time I think setting up a long running reputation weighted reputation clearing house would be amazing. (And if nothing else at least something like this could emerge from all the crypto-staking-reputation research. See Polkadot and Ethereum's plans about keeping folks honest with bounties (reverse staking?) for example.)

Well actually Spoofing on a private network is trivial, but in TCP over a public network, it's another story entirely and it's not simple at all. UDP can be easily spoofed though, hence we do not treat reports in the same way so far. Beyond this, there is eventually BGP spoofing, but funny enough, CrowdSec could detect them, provided you have logs. It should be fairly easy to track in terms of behavior.

You wouldn't have to spoof. I said seed, not spoof - you could just spin up a bunch of servers, connect it to the crowdsourced security networks, and issue false claims that IP address X has been attempting to break into your network, and suddenly block X from their own servers.

Well we have a consensus system that's quite advanced to avoid poisoning and false positives. To put it short, all members have a Trust rank, only TR1 can publish an IP without counter verification, and only if it doesn't shoot a Canari from our whitelist of IPs. TR1 mean perfect accurate reports for 1+ year. All other TR level can partake but need counter verification from either our own honeypot network or other TR1 peers before being integrated. There is also an AI that will be trained soon to confirm false negatives and extract more complexe patterns.

Thanks for the details!

So basically anyone joining the network for the next year sits in limbo, the network is not capable of catching more "bad IPs" for that year, because any report by new members requires cross-verification by the original nodes/honeypots.

This seems pleasantly conservative. Also, is there a way for nodes to lose trust rank? (How will the network find out if a TR1 node is reporting false negatives?)

this is accounted for. By default you have a whitelist containing local lan IP ;)

Yeah I have 10+ machines with fail2ban configured and 0 of them on my LAN though.

well just whitelist your Public IPs or use a combo of IPset & port knockd. Works fine for me for variable IPs.

LAN addresses, eh? People do still use internet addressing on our networks despite the consumer CPE vendors increasingly trying to sell you NAT stockholm syndrome :)

the machines i'm ruining fail2ban on are on public networks, not my LAN.

You don't use static IPs for managing systems?

There is some discussion in the diquis about them providing a server you can run yourself. So if you have a fleet of servers you don't have to share with crowdsec but instead can share only with yourself (my read, perhaps a hybrid)

yepn indeed, we call it private sharding or private consensus. Far on the roadmap (4 months), but nevertheless, the team is thinking about it. You could also include or exclude some Geographics for ex if you don't trust a country or have a private consensus between only your own machine. If you are, say Morgan Stanley, you may be attacked only you by some machines and the crowd wouldn't know. But all your servers teaming together will see it.

To me the main value proposition would be blocking anyone who was observed attacking one of my servers or even a honeypot. I'm not opposed to sharing that data as far as GDPR allows, but receiving block lists is something I would be a lot more careful about.

I have the opposite feeling.

I don’t see why we need to replace fail2ban, but I find the idea of a global repository of bad IPs and reputation very interesting.

I can think of a few: https://www.cvedetails.com/vulnerability-list/vendor_id-5567...

Fail2ban is a bit shaky when it transitions to Python3.

Hasn’t pass the pylint,?futurize, or 2to3 by much.

But if the maintainers are willing, I can do those things.

Not sure what you mean by shaky, fail2ban has been working under Python 3 for quite a while. (See for instance fail2ban on Ubuntu 16.04 depending on python3[0].) People definitely would have noticed.

I saw your issue[1] on GitHub, and that's just a misunderstanding of how fail2ban was ported to Python 3. fail2ban uses the use_2to3 feature of setuptools.setup to do automatic translation upon installation.[2] While I do wish the code runs under Python 3 as is, what users actually run is totally working code.

[0] https://packages.ubuntu.com/xenial/amd64/fail2ban

[1] https://github.com/fail2ban/fail2ban/issues/2853

[2] https://github.com/fail2ban/fail2ban/blob/960e30cfcdae7e2c81...

Yes. It may be totally working code.

But it wasn’t written cleanly as many tools would attest.

Those tools tell you if you comply with some persons style choices, and maybe catch a few potential problems. That says close to nothing about cleanliness or correctness. They're for internal use, not for making external measurements of code.