News for hackers. Stuff that matters.

A comment on I protect my forgejo instance from AI Web Crawlers.

Re:I protect my forgejo instance from AI Web Crawlers by BLKNSLVR on Monday December 22, 2025 @07:10AM (#46353532)
I really don't know how effective my little system would be against these scrapers, but I've setup a system that blocks IP addresses if they've attempted to connect to ports on my system(s) behind which there are no services, and therefore their connections must be 'uninvited', which I classify as malicious.

Since I do actually host a couple of websites / services behind port 443, it means I can't just block everything that tries to scan my ip address at port 443. However, I've setup Cloudflare in front of those websites, so I do log and block any non-Cloudflare (using Cloudflare's ASN: 13335) traffic coming into port 443.

I also log and block IP address attempting to connect on port 80, since that essentially deprecated.

This, of course, does not block traffic coming via the DNS names of the sites, since that will be routed through Cloudflare - but as someone mentioned, Cloudflare has its own anti-scraping tools. And then as another person mentioned, this does require the use of Cloudflare, which is a vast centralising force on the Internet and therefore part of a different problem...

I don't currently split out a separate list for IP addresses that have connected to HTTP(S) ports, but maybe I'll do that over Christmas.

This is my current simple project: https://github.com/UninvitedActivity/UninvitedActivity

Apologies if the README is a bit rambling. It's evolved over time, and it's mostly for me anyway.

P.S. I always thought it was Yog Sothoth (not Sototh). Either way, I'm partial to Nyarlathotep. "The Crawling Chaos" always sounded like the coolest of the elder gods.

[ Parent | on HN ]
Re:I protect my forgejo instance from AI Web Crawlers by ewpratten on Monday December 22, 2025 @07:24AM (#46353607)
Regarding the Cloudflare part of this, I’d recommend taking a look at “Authenticated Origin Pulls”. It lets you perform your validation at the TLS layer instead of doing it with IP ACLs if that interests you.
[ Parent | on HN ]
Re:I protect my forgejo instance from AI Web Crawlers by mmarian on Tuesday December 23, 2025 @02:33AM (#46363232)
Some scrapers/scanners use residential IPs. Aren't you worried you'll end up blocking legitimate traffic?
[ Parent | on HN ]
Re:I protect my forgejo instance from AI Web Crawlers by BLKNSLVR on Wednesday December 24, 2025 @10:53AM (#46376606)
Two points:

1. The websites I run get so little traffic it doesn't matter. They're mostly for my own entertainment / experimentation.

2. If they're allowing their IP address to be used for pricing scrapers then I consider that within the blurry definition of malicious anyway.

I don't mind if you disagree with me on point #2, and I grant that if I was running some super popular web service, maybe my tune would change.

I have four tiers of scanning paranoia, so I can ramp up and down if need be (I'm not sure if that's documented in GitHub though...)

[ Parent | on HN ]
Re:I protect my forgejo instance from AI Web Crawlers by mmarian on Wednesday December 24, 2025 @12:55PM (#46377651)
Fair enough. Let me know if you end up documenting it in GitHub somewhere, I'm curious what people set up.
[ Parent | on HN ]
Re:I protect my forgejo instance from AI Web Crawlers by alsetmusic on Thursday December 25, 2025 @02:28PM (#46386504)
Haha, looks like you use sshpass [0], which I only discovered by accident a couple of months ago. I wasn't able to get the current version of it to work for some reason, but I was able to debug it into a working state by combining bits from the current version and an earlier version.

[0] https://github.com/UninvitedActivity/UninvitedActivity/blob/...

[ Parent | on HN ]