Can you please write more details about "I block many AI crawlers from accessing code and photos"? The bots are trying to access your nextcloud instance? I'm also self hosting a few services, including nextcloud.
No, not nextcloud, it's the photos on my website. They are CC-BY-NC-ND-4.0 licensed, which genAI doesn't respect in any form.
I added these in nginx.conf:
map $http_user_agent $blocked_user_agent {
default 0;
"~*AI2Bot" 1;
"~*AI2Bot-Dolma" 1;
"~*Amazonbot" 1;
"~*anthropic-ai" 1;
"~*anthropic.com" 1;
"~*Applebot" 1;
"~*Applebot-Extended" 1;
"~*AwarioBot" 1;
"~*AwarioRssBot" 1;
"~*AwarioSmartBot" 1;
"~*Bytespider" 1;
"~*CCBot" 1;
"~*ChatGPT-User" 1;
"~*ClaudeBot" 1;
"~*Claude-Web" 1;
"~*cohere-ai" 1;
"~*cohere-training-data-crawler" 1;
"~*DataForSeoBot" 1;
"~*Diffbot" 1;
"~*DuckAssistBot" 1;
"~*FacebookBot" 1;
"~*FriendlyCrawler" 1;
"~*Googlebot-Extended" 1;
"~*Google-CloudVertexBot" 1;
"~*Google-Extended" 1;
"~*GoogleOther" 1;
"~*GoogleOther-Image" 1;
"~*GoogleOther-Video" 1;
"~*GPTBot" 1;
"~*iaskspider/2.0" 1;
"~*ICC-Crawler" 1;
"~*ImagesiftBot" 1;
"~*img2dataset" 1;
"~*ISSCyberRiskCrawler" 1;
"~*Kangaroo Bot" 1;
"~*Meltwater" 1;
"~*Meta-ExternalAgent" 1;
"~*Meta-ExternalFetcher" 1;
"~*OAI-SearchBot" 1;
"~*Omgili" 1;
"~*Omgilibot" 1;
"~*openai.com" 1;
"~*PanguBot" 1;
"~*peer39_crawler" 1;
"~*PerplexityBot" 1;
"~*PetalBot" 1;
"~*Scrapy" 1;
"~*Seekr" 1;
"~*SemrushBot" 1;
"~*SemrushBot-OCOB" 1;
"~*Sentibot" 1;
"~*Sidetrade indexer bot" 1;
"~*Timpibot" 1;
"~*TurnitinBot" 1;
"~*VelenPublicWebCrawler" 1;
"~*webmeup-crawler.com" 1;
"~*Webzio-Extended" 1;
"~*YouBot" 1;
}
location / {
if ($blocked_user_agent) {
access_log /var/log/nginx/blockedbot.log ncsa;
return 401;
}
Thanks!
Re:Moving away from US cloud services
by dracotomes on Tuesday March 18, 2025 @06:37AM
(#43397789)
I self host a mail server too. Static IPs are not available for a residential connection where I live which makes PTR and SPF records hard.
What I've been doing is using Oracle's "always free" VM to host VyOS as an ingress/egress router. It's been pretty easy to set up the mail server itself using iRedMail.
I've had some problems with spamhaus and outlook because that whole IP range is not trustworthy but otherwise it's just worked (Spamhaus was nice enough to put me on a whiltelist).
I know hosting at Oracle is not moving away from US cloud services but I set this up two years ago and this could be achieved using any cheap or free VPS.
If you have a fixed IP with a good reputation then 95% of your email problems are solved. If you use a client (like the iphone email client), your setup is really simple (as in set it up once and forget about it). Providers will also try re-sending email if it doesn't reach you, so availability is not that big of a deal.
That being said, the major issue starting up is having that highly reputable ip/domain.
FAQ:
Yes, you can run self hosted mail.
Yes, it's complex.
No, it's not hard, but it takes time, as it is complex - if you want to understand it. If not, go for something like https://maddy.email/
Yes, I have a fixed IP address with a reverse DNS entry configured.
My ISP is zen.co.uk.
Yes, I have reverse DNS, DKIM, DMARC, SPF, even mta-sts.
No, I'm not switching from XMPP to Matrix - it's too server heavy, XMPP is more client oriented, which is my preference.
I know about the UK's online safecty act, I used https://onlinesafetyact.co.uk/ra_my_self_hosted_single_user_... as a template to create mine. I think the act itself is basically useless and just paperwork, but something along it's ideas is actually needed.
I block many AI crawlers from accessing code and photos, as eg. Claude is so aggressive that it's code crawlers makes my system sluggish, and I have no will to let anyone use my photos beyond printing them for their wall as decoration.