CodeBytes

The bot I had was using unique IPs for each request. Some were from cloud providers but most were just random residential ISPs. I couldn't see any obvious connections so rate limiting would've had to be a global rate limit.

Similar to the one SQLite had: https://www2.sqlite.org/forum/forumpost/7d3eb059f81ff694?t=h

Each IP only makes ~1 request though so easy to detect after the fact.

I guess they will run out of IPs at some point so maybe if I had logged each one forever and shown a challenge only to them, it would have fixed it eventually. Just depends how big their pool of IPs is.

I'm not 100% sure but I think links. There's a bunch on the history and revision pages. Yeah, the diff URL has two revision ID's as parameters.

I did try removing some of the links without success. I guess once they have them they just keep checking.

I think people get annoyed when it's suggested they spend time optimising or even re-writing their websites to handle high traffic loads just to cater to AI bots ripping their content.

It's also not always easy to do. I run a small wiki which is fairly optimised, nearly every page manages at least ~3k rps on a small VPS. The only exception is the diff page which is ~150 rps. Optimising that while still giving good output isn't that easy, but the wiki doesn't have many users so that would be fine if it wasn't for the AI bots.

The AI bots ignore robots.txt and were initially hitting the site with ~1k rps crawling every combination. Even that would be manageable as there's currently ~150,000 combinations, except they kept re-crawling the whole lot each day. The server could manage it but it was a massive waste of resources.

They were using residential IPs and only sending 1 request from each IP making it impossible to block. In the end I gave up and put a Cloudflare challenge in front of it. I don't want to use Cloudflare but the alternative is forcing users to login to view diffs or remove them entirely.

This feels like it's becoming less and less true, good quality items are becoming so expensive now or very hard to find.

I do think it is still very true for tools though. It's nearly always worth getting decent ones, they nearly give better results or are easier to use and last so much longer.

It likely is AI scrapers essentially doing a DDoS. They use separate IPs (and vary the UA) to prevent blocking.

I have a site which is currently being hit (over 10k requests today) and it looks like scrapers as every URL is different. If it was a DDoS, they would target costly pages like my search not every single URL.

SQLite had the same thing: https://sqlite.org/forum/forumpost/7d3eb059f81ff694 As have a few other open source repositories. It looks like badly written crawlers trying to crawl sites as fast as possible.

Unless Trump wants to start WW3 the US won't be able to do anything, and even if he did start WW3, Europe would be able to destroy those bases.

The whole point of them was to give the US influence while improving US security. Given Europe can't trust Trump will come to their aid, they won't give the US as much influence over Europe.

The Lemmy homelab community is up to 1.67k subscribers (https://lemmy.ml/c/homelab) so while people may not be joining a specific homelab instance, they are creating accounts somewhere and joining the community.

Having said that, it doesn't seem very active at the moment which isn't a good sign. Granted, I have no idea how active the subreddit is to compare.

The selfhosted community on Lemmy (https://lemmy.world/c/selfhosted) is up to 12.5k subscribers and seems fairly active so it's possible there might be enough people who care this time for it to work.

That doesn't seem to be true though. There are multiple countries outside the EU that have an adequacy decisions regarding their privacy laws like: Japan, South Korea, Canada, UK, Isreal, etc. They can host EU data without issues.

The only reason the privacy shield agreement was thrown out was due to lack of safe guards from US intelligence.

Even without the privacy shield, US companies would still be able to store EU data in a country with an adequacy decision if it wasn't for the CLOUD act. This seems more to do with US law wanting access to EU data.

There was a very recent case which, at least for some types of data, found Cloudflare is not adequate: https://edpb.europa.eu/news/national-news/2021/census-2021-p...

It looks like the court decided SCCs were not sufficient as Cloudflare is subject to US surveillance laws so they wouldn't be able to provide adequate guarantees.

Fastmail Professional allows 80,000 inbound, 16,000 outbound per day [1].

[1]: https://www.fastmail.help/hc/en-us/articles/1500000277382-Ac...

Neat idea. What's the criteria? Just it seems to be missing Google Code. Had a look on Wikipedia and seems to be a lot missing from that too: https://en.wikipedia.org/wiki/List_of_Google_products#Discon...