News for hackers. Stuff that matters.
Re:Brute.Fail Watch brute force attacks in real time by ChuckMcM on Friday June 02, 2023 @05:34PM (#36171049)
So funny story, for a while I worked on a 'reverse' exploit. Which is to say morphing the response from ssh to the client with large malformed packets. The idea was to crash the client making the request. In my case I found these attacks would have like 6 to 10 attempts from the same source address. By time stamping the requests, I could evaluate if the next attack from the same address came more quickly or more slowly. I then had my server "morph" the return payload somewhat randomly and keep the three responses that caused the most slowdown. When I got to 100 variants that had "won" this selection criteria I took the three best and started over from there. After a couple of months of this I finally got a response where after one request I would not get a second.

I felt extremely pleased with myself for about another month, and then my server address got hit with a massive DDOS attack (for me anyway) over my 6MBPS DSL line. So clearly I had hit a nerve somewhere :-). Anyway, I moved my server to a different address and used fail2ban to just note source IPs and put them into the IP tables as banned addresses. That works great and hasn't resulted in the same sort of drama as last time.

Re:Brute.Fail Watch brute force attacks in real time by mike_hock on Friday June 02, 2023 @05:40PM (#36171135)
You still got the response packets?
Re:Brute.Fail Watch brute force attacks in real time by sneak on Friday June 02, 2023 @05:38PM (#36171102)
Disable password authentication and fail2ban becomes completely unnecessary.
Re:Brute.Fail Watch brute force attacks in real time by nico on Friday June 02, 2023 @05:43PM (#36171169)
You still get the attacks and they show in the logs

Also, it’s not just ssh, there are brute force attacks for pretty much any service that you run

Our logs show automated attempts to run exploits on our web servers everyday

Re:Brute.Fail Watch brute force attacks in real time by BrandoElFollito on Friday June 02, 2023 @05:52PM (#36171268)
This. I really do not understand why people use fail2ban when the threat is somewhere else.

It won't stop a ddos but will certainly, at some point, prevent you from logging in.

Re:Brute.Fail Watch brute force attacks in real time by thelopa on Friday June 02, 2023 @06:07PM (#36171439)
I disable password authentication and use fail2ban. It's unlikely they will be able to brute force my key, but no server is perfect. sshd might be compromised one day. I’d rather have an extra layer of defense just in case.
Re:Brute.Fail Watch brute force attacks in real time by wkat4242 on Friday June 02, 2023 @08:35PM (#36172559)
It's impossible they will brute force your key if you have a decent length.

While I'm sure it is possible for some (mainly government) actors to brute force keys, I'm also sure these do not include the same low-hanging-fruit vandals blasting brute force attacks. And I'm also pretty sure you're not one of the select targets of these highly advanced actors.

A vulnerability in sshd is indeed possible and happens once in a while. Fail2Ban won't stop this though because a known exploit will let them through on the first attempt.

I personally view fail2ban more as nuisance control. Minimizing the log crap, the wasted CPU resources by the failed handshakes. It's not really a security protection.

Re:Brute.Fail Watch brute force attacks in real time by sophacles on Friday June 02, 2023 @10:20PM (#36173083)
Unless the maintainer of your distro's ssh package accidentally introduces an error that reduces the number of possible keys to, say - 32,767 total possible keys.[1] That's a brute-forcible number of keys that fail2ban would help mitigate.

(1: https://research.swtch.com/openssl)

Re:Brute.Fail Watch brute force attacks in real time by rerdavies on Saturday June 03, 2023 @06:01AM (#36175234)
Or the NSA subverts a cryptographic standard in order to produce predictable seeds for cryptographic random-number generators that are used to produce private keys. [1]

(1: https://www.bbc.com/news/technology-24048343 )

Re:Brute.Fail Watch brute force attacks in real time by drpixie on Sunday June 04, 2023 @04:37AM (#36184048)
INCONCEIVABLE!

You keep using that word. I do not think it means what you think it means.

Re:Brute.Fail Watch brute force attacks in real time by theamk on Friday June 02, 2023 @06:04PM (#36171407)
It keeps the logs cleaner. If you either don't look at the logs at all, or have fancy log analysis systems, then it does not matter. But if you are in the middle, and just manually look at the logs every once in a while, this would be a great help.

If you are logging in via ssh, the chances of being locked out arr low - using password auth is a bad idea, and once you set up ssh keys, the connection will always succeed. And in case of rare event like new system setup, you can always ssh via some other system, -J is great for that.

Re:Brute.Fail Watch brute force attacks in real time by admax88qqq on Friday June 02, 2023 @06:20PM (#36171557)
> the connection will always succeed.

Not from my experience. If you have too many keys and certain ssh agents like gnome keyring don't pick up the key intelligently and will try a all the keys in some order often resulting in the server giving rejecting you due to too many failures.

Re:Brute.Fail Watch brute force attacks in real time by theamk on Friday June 02, 2023 @07:23PM (#36172096)
Yeah, this is annoying.

But you don't live with it -- you either move the extra keys to subdir, so that gnome keyring does not pick it; or use "IdentitiesOnly yes"/"IdentityFile foo" in .ssh/config to restrict certain hosts to certain keys (and yes, those work with ssh agent caching too).

I know many people just don't care about working tool, and tolerate the pain, but hopefully if someone knows enough to setup fail2ban, they should also be able to setup ssh config. Especially since reliable ssh connections is such a high quality of life improvement.

Re:Brute.Fail Watch brute force attacks in real time by oehpr on Friday June 02, 2023 @07:36PM (#36172187)
Yes I've hit this as well! Such an annoying behaviour.

However I think it's a good habit to make records in `~/.ssh/config` for each of your servers anyway just to keep tabs what, where, who, and with what keys.

Re:Brute.Fail Watch brute force attacks in real time by BrandoElFollito on Saturday June 03, 2023 @05:31AM (#36175092)
> It keeps the logs cleaner.

I move my port somewhere else to reduce the clutter

> manually look at the logs every once in a while

I imagine that this is more by curiosity than anything else (which is a perfectly normal reason to do that - I do it from time to time just because). For logs analysis and alerting I have automated systems.

> using password auth is a bad idea

It is not if the password is correct, security speaking.

Re:Brute.Fail Watch brute force attacks in real time by bandrami on Friday June 02, 2023 @06:51PM (#36171860)
20 years ago port-knocking was supposed to solve this issue for good but it seems to have been never really been taken up. I'm not sure why.
Re:Brute.Fail Watch brute force attacks in real time by NavinF on Friday June 02, 2023 @07:38PM (#36172206)
Because it doesn't solve any real problems if you have ssh password login disabled and filter out login failures from logs
Re:Brute.Fail Watch brute force attacks in real time by bombela on Friday June 02, 2023 @08:24PM (#36172482)
With port knocking or a simple as having SSH on a non-standard port, the connection request stops before even opening the TCP connection. That's less load on the system, less logs, less writing to storage, etc. Less of what you don't want must surely be preferable.
Re:Brute.Fail Watch brute force attacks in real time by 9dev on Saturday June 03, 2023 @03:49AM (#36174592)
…all for the cost of vastly increased complexity. You’re just micromanaging your pets there. If the load of failed login attempts or the log writes brings your servers down, you have a whole lot of other problems to take care of.
Re:Brute.Fail Watch brute force attacks in real time by NavinF on Saturday June 03, 2023 @04:10AM (#36174688)
Are we talking about ssh on an ESP32 or a 2U server? You'd saturate my transit port before I noticed the load on my server or the logs that I filter by default.

btw TCP is cheap compared to public key crypto

Re:Brute.Fail Watch brute force attacks in real time by corn13read2 on Friday June 02, 2023 @08:15PM (#36172425)
This is really only effective for password based logins, but most logins these days are key based
Re:Brute.Fail Watch brute force attacks in real time by bandrami on Saturday June 03, 2023 @12:03AM (#36173592)
No, it cuts the DOS resource usage by a factor of 4 because a TCP connection is never opened.
Re:Brute.Fail Watch brute force attacks in real time by wkat4242 on Friday June 02, 2023 @08:49PM (#36172626)
Port knocking as it's usually done is easily sniffed. Perhaps using a dynamic TOTP-like time based seed to constantly rotate the ports might help. But it sounds overly complex.

It feels very like a "key under the third plant on the right" kinda thing. Not a solid security measure.

Re:Brute.Fail Watch brute force attacks in real time by matheusmoreira on Friday June 02, 2023 @11:38PM (#36173468)
Single packet authorization. It's like the server is not even there unless you send a cryptographically signed packet

https://cipherdyne.org/fwknop/docs/SPA.html

Re:Brute.Fail Watch brute force attacks in real time by wkat4242 on Saturday June 03, 2023 @08:24AM (#36176063)
Nice solution. I hadn't heard of it. I immediately thought of replay attacks because it's a one-way protocol but it looks like they mitigated those as well.
Re:Brute.Fail Watch brute force attacks in real time by 0cVlTeIATBs on Friday June 02, 2023 @11:38PM (#36173472)
Moxie Marlinspike made something to address the simple knock server's problems.

https://archive.is/MZKkb

Re:Brute.Fail Watch brute force attacks in real time by Gordonjcp on Saturday June 03, 2023 @05:17AM (#36175024)
This is fine, though. "Security through obscurity is not security" but moving your SSH port to something not 22 will utterly eliminate brute force attacks.

It's too much bother to go find it, and the bozos will just move on to the next machine with port 22 open.

Re:Brute.Fail Watch brute force attacks in real time by wkat4242 on Saturday June 03, 2023 @05:46AM (#36175162)
That's still not really security but just a nuisance mitigation IMO :)
Re:Brute.Fail Watch brute force attacks in real time by Gordonjcp on Saturday June 03, 2023 @05:51AM (#36175189)
It's both, really. If you're not getting scanned, you've reduced an attack surface, and that can only be good.
Re:Brute.Fail Watch brute force attacks in real time by wkat4242 on Saturday June 03, 2023 @08:19AM (#36176029)
Well, I view it as hiding an attack surface. It's still there, just harder to find.

But I know I'm a bit of an absolutist on security.

Re:Brute.Fail Watch brute force attacks in real time by Gordonjcp on Saturday June 03, 2023 @06:12PM (#36181020)
You put a lock on your bike.

But you also put it in the shed, and lock the shed.

Re:Brute.Fail Watch brute force attacks in real time by jlund-molfese on Saturday June 03, 2023 @09:23AM (#36176472)
Nuisance mitigations are part of security too! Fewer irrelevant notifications makes it more likely you’ll notice when something really is a problem.

It’s like how an adversary might launch a DDoS attack at the same time as they exploit a SQL injection vulnerability to exfiltrate credit card information. Filling up logs and alerts overwhelms the blue team and makes it harder to notice the quieter, but more dangerous attack.

Re:Brute.Fail Watch brute force attacks in real time by oh_sigh on Saturday June 03, 2023 @10:24AM (#36176941)
Security through obscurity actually is security, and is perfectly valid to use with a defense in depth strategy. The problem is when obscurity is the only defense.
Re:Brute.Fail Watch brute force attacks in real time by bigkm on Saturday June 03, 2023 @12:54PM (#36178244)
I agree, these attacks are looking for systems that have pretty default security, and by running on a different port you avoid all this automated chaos because you're non default now. Like scam emails with typos, a way to filter out the naïve people.
Re:Brute.Fail Watch brute force attacks in real time by tredre3 on Friday June 02, 2023 @07:07PM (#36171980)
> This. I really do not understand why people use fail2ban when the threat is somewhere else.

Fail2ban keeps my log short enough that I can review them daily, I don't have to sift through thousands of login attempt.

> It won't stop a ddos but will certainly, at some point, prevent you from logging in.

Yup, losing my key and having no password access will do that.

Re:Brute.Fail Watch brute force attacks in real time by jrockway on Friday June 02, 2023 @07:42PM (#36172221)
I'm not sure manual log scanning is all that interesting. You probably want to produce a list of successful logins and review that regularly.
Re:Brute.Fail Watch brute force attacks in real time by wkat4242 on Friday June 02, 2023 @08:47PM (#36172619)
If an attacker manages a succesful login you've already lost. Better to catch them in the act while they're scoping you out.
Re:Brute.Fail Watch brute force attacks in real time by jrockway on Friday June 02, 2023 @08:59PM (#36172665)
If your SSH key leaks you're not going to have a warning. All you'll see is a login from yourself that you don't remember.

I am pretty sure we turned off password authentication like 10 posts up this thread.

Re:Brute.Fail Watch brute force attacks in real time by elesiuta on Friday June 02, 2023 @11:26PM (#36173407)
This is why I require both a private key and a password.

I have fail2ban configured to block IPs with invalid private keys after a couple attempts, and if the key is valid to email me and rate limit invalid password attempts.

This gives a more than sufficient warning if my key leaks which is already very unlikely, and this just makes it much more unlikely for both to be compromised, and only took an extra 5 minutes to configure.

Re:Brute.Fail Watch brute force attacks in real time by phanimahesh on Saturday June 03, 2023 @04:34AM (#36174821)
How do you configure emails on successful logins? Can you share your config, sufficiently anonymized?
Re:Brute.Fail Watch brute force attacks in real time by elesiuta on Saturday June 03, 2023 @02:07PM (#36178998)
I created a jail for fail2ban with

  logpath = /var/log/auth.log
and for the filter I use

  failregex = .*Connection closed by authenticating user [a-z_]([a-z0-9_-]{0,31}|[a-z0-9_-]{0,30}\$) <HOST> port [0-9]* \[preauth\]
and the emails are just cron with a python script that checks /var/log/fail2ban.log for any new Found|Ban|Unban IPs and sends them using smtplib

If you like I can share the full config files but the rest isn't too interesting nor different from what is here https://www.fail2ban.org/wiki/index.php/MANUAL_0_8#Jails

Re:Brute.Fail Watch brute force attacks in real time by BrandoElFollito on Saturday June 03, 2023 @05:28AM (#36175074)
I move my port to an uninteresting place and it considerably reduces the amount of logs. If this was a nuisance I would filter out the messages

> Yup, losing my key and having no password access will do that.

You can also lose your password. Or forget it if you know it by heart. But in any case you can use a password instead of a key - it just needs to be good enough (= long and not in cracking dictionaries)

Re:Brute.Fail Watch brute force attacks in real time by tjoff on Saturday June 03, 2023 @06:16AM (#36175308)
I really don't understand why people use keys when the threat is somewhere else.
Re:Brute.Fail Watch brute force attacks in real time by BrandoElFollito on Saturday June 03, 2023 @06:57AM (#36175524)
Assuming that your comment is serious, where is the threat in your opinion?
Re:Brute.Fail Watch brute force attacks in real time by tjoff on Saturday June 03, 2023 @06:58AM (#36175534)
Bad passwords.
Re:Brute.Fail Watch brute force attacks in real time by BrandoElFollito on Saturday June 03, 2023 @07:01AM (#36175551)
Then I do not understand. Keys are the answer to bad passwords (among other things)
Re:Brute.Fail Watch brute force attacks in real time by tjoff on Saturday June 03, 2023 @07:17AM (#36175658)
It is one answer. Good passwords is another.
Re:Brute.Fail Watch brute force attacks in real time by XorNot on Friday June 02, 2023 @07:50PM (#36172270)
I prefer libshield but one thing I've found is that annoyingly sshd didn't use PAM to check if the login user is valid apparently (so it never fires when only using keys).
Re:Brute.Fail Watch brute force attacks in real time by wkat4242 on Friday June 02, 2023 @08:33PM (#36172551)
Not really. A lot of these bots are super duper dumb and continue slamming your server with handshake attempts. Every handshake is wasted CPU resources (and asymmetric crypto isn't cheap).
Re:Brute.Fail Watch brute force attacks in real time by scrps on Friday June 02, 2023 @10:09PM (#36173012)
And/or run sshd exposed only to a wireguard or tailscale interface.
Re:Brute.Fail Watch brute force attacks in real time by seized on Friday June 02, 2023 @11:34PM (#36173448)
Not for other services... I have fail2ban parsing my NVR logs. Wrong password three times and it permanently blocks the IP on my Opnsense firewall.
Re:Brute.Fail Watch brute force attacks in real time by nvarsj on Saturday June 03, 2023 @05:59AM (#36175227)
And change the sshd port. I use my birth year. 0 brute force attempts in the logs after that.
Re:Brute.Fail Watch brute force attacks in real time by DougMerritt on Friday June 02, 2023 @05:56PM (#36171303)
Nice. Also an illustration that its good to establish one's inner motive first, like here might range from a desire for simple solutions, to another dimension of drama, which might indeed be desirable sometimes, for entertainment or for snapshots to use in a book one is writing. :)
Re:Brute.Fail Watch brute force attacks in real time by lgats on Friday June 02, 2023 @08:41PM (#36172590)
Another fun one is sending a redirect to a gzip bomb or extremely large file (i.e. speedtest download file) for when certain non-existant URLs are requested, i.e. /wp-login.php on a non-wordpress site.
Re:Brute.Fail Watch brute force attacks in real time by KomoD on Friday June 02, 2023 @10:29PM (#36173140)
Earlier this month I was downloading top 1 million websites' robots.txt and some guy just served 100mb of whitespace :/
Re:Brute.Fail Watch brute force attacks in real time by pnpnp on Friday June 02, 2023 @10:36PM (#36173178)
I love it!
Re:Brute.Fail Watch brute force attacks in real time by hayley-patton on Friday June 02, 2023 @10:42PM (#36173213)
Once I was trying to scrape from some internal API, and made a request that used parameters the real client wouldn't make. I didn't expect curl to scream "FUCK YOU <some name>" at me. Presumably the name belonged to someone who scraped it before.
Re:Brute.Fail Watch brute force attacks in real time by KomoD on Saturday June 03, 2023 @08:29AM (#36176093)
LOL, that is great.
Re:Brute.Fail Watch brute force attacks in real time by gitgud on Saturday June 03, 2023 @12:14PM (#36177857)
Is it legal to host a gzip bomb? Seems like a malicious file to serve up…
Re:Brute.Fail Watch brute force attacks in real time by bytesandbots on Saturday June 03, 2023 @12:37PM (#36178102)
It can cause your site to end up on Google's safe Browsing black list which can be a death sentence for a business. Google has automated process for identifying malware and black list such websites. Almost all browsers use this list to warn users. This is why it is also dangerous to host anonymous uploaded files even for a short time. https://news.ycombinator.com/item?id=25802366
Re:Brute.Fail Watch brute force attacks in real time by michaelcampbell on Saturday June 03, 2023 @03:12PM (#36179625)
> It can cause your site to end up on Google's safe Browsing black list

If it's a file that nothing on your site links to, and doesn't really "exist" how would google ever index it? Especially if you put in as a deny in robots.txt, which as far as I'm aware, Google honors.

Re:Brute.Fail Watch brute force attacks in real time by quickthrower2 on Friday June 02, 2023 @09:30PM (#36172825)
Distributed valgrind
Re:Brute.Fail Watch brute force attacks in real time by api on Friday June 02, 2023 @09:38PM (#36172867)
Try negotiating ssh compression and then sending terabytes of compressed zeroes. Might work for a badly implemented scanner.
Re:Brute.Fail Watch brute force attacks in real time by KomoD on Friday June 02, 2023 @10:04PM (#36172994)
Semi-related, I've just started serving large files to webserver scanners, some are so poorly made that they just download the entire file
Re:Brute.Fail Watch brute force attacks in real time by Gordonjcp on Saturday June 03, 2023 @05:10AM (#36174986)
Please tell me it's a massively upscaled hello.jpg you send...
Re:Brute.Fail Watch brute force attacks in real time by heyoni on Saturday June 03, 2023 @11:05AM (#36177244)
Details!

Seriously this is both hilarious and intriguing and deserves a long form blog post or something.

Re:Brute.Fail Watch brute force attacks in real time by myself248 on Saturday June 03, 2023 @01:16PM (#36178493)
Seriously, this should be standard practice. Fail2evil...
Re:Brute.Fail Watch brute force attacks in real time by fireant on Sunday June 04, 2023 @12:14AM (#36183039)
Wow that's a cool evolutionary algorithm in practice! I'm actually surprised it worked, especially with the amount of fuzzing I'd expect ssh ecosystem to receive (well perhaps the servers do, but not the clients haha).
Re:Brute.Fail Watch brute force attacks in real time by Jenda_ on Sunday June 04, 2023 @09:07AM (#36185312)
MySQL has a weird feature, by default enabled in many clients, that the server can request a file from the client.

I have requested /etc/shadow, cracked the hash for root password, and ssh'd back into the botnet node that was bruteforcing passwords. I then shared the information with the webhoster where the botnet was running and a local infamous antivirus company (Avast before it was leaked that they are evil) and got a t-shirt.

https://www.abclinuxu.cz/blog/jenda/2019/2/exploiting-mysql-...

Re:Brute.Fail Watch brute force attacks in real time by mdaniel on Sunday June 04, 2023 @01:56PM (#36187758)
Outstanding job! My opinion of mysql continues to improve :-/

As a curiosity, I found the submission link in the comments on that thread https://news.ycombinator.com/item?id=19305823