If they’re ipv6 address wouldn’t they be safe to block across large ranges?
It is also trivially “easy” to get past reCAPTCHA, but it costs more. My guess is that a domain name checker tool isn’t worth the cost per request to bypass reCAPTCHA (approximate 0.02 cents per session)
It’s hard to “throttle” a single isolated request from a lone IP.
There's only so much you can really do when your underlying resource is so limited. Luckily the value of the query is lower than the cost of a recaptcha solve, so the attackers moved on to some other target.
Ironically I could now turn off the endpoint protection (or have it responsive to traffic load), until the attackers return. I shall not go into too many details, no need to give people a map.
I first thought it was something like "quadrillion bytes per second" or some newer over-the-top data measurement :)
The main idea here is that at some point just leaving the scrape running in a way that didn't overwhelm my backend would have resulted in me not caring enough to do that. But now they get nothing. Even if you're borrowing bandwidth and not paying, you should be a good neighbor is all.
The webmaster doesn’t need to worry about it, the anti-bot services handle who gets what difficulty of challenge. But the webmaster can specify whether they’d like to be more or less strict/difficult than usual.
Though thinking about it, I wonder if there is a hybrid: start with a difficulty that's just a few seconds for CPU/WebCrypto and ramp up quickly, but also support WebGPU where possible so that web users on abusive connections may still succeed? I am not sure though, I guess this depends on the feasibility of using WebGPU and etc.
I commented in the past about it:
"At first glance, yes, we can create intentionally expensive computations without relying on a blockchain, that would serve the same purpose. In reality we cannot. Special computer hardware (ASICs) could generate much cheaper PoW annotations than general purpose computers, and sell it to spammers. Blockchain economic incentives ensure that ASICs will be used by the miners first and foremost."
So instead, what, you want people to buy BTC and send small amounts of it to websites?
Problem is, given varying income levels across society and across the world, one person's expensive micropayment is another person's almost free micropayment.
Very true, that micropayments could vary on their relative cheapness across the global population. Let's put a number, is 0.01 cent affordable by most people on the planet, for every http request?
An obscure coin my users cannot purchase through major exchanges is a really poor solution. Sorry, that makes no sense.
But again, economic incentives need to work their way into the system.
In case you run out of proof_of_burn a friend of yours might send you just a cent, and you are good to go, for one hundred thousand http requests more. There is no need for a normal person, to hold more than a handful of dollars for every year's internet use.
That renders exchanges almost useless. Not totally useless, but much less relevant than they are today.
I personally wouldn't care less, if there is one bitcoin/blockchain which reaches that sweet spot, or if they are a hundred, including litecoin, ripple etc. PoW is meant to be used for practical reasons. Blockchain however constitutes an economic system, of suppliers-miners and consumers-users, it is more than just a software program. It will evolve in the future, and it requires some crucial time.
For the moment there no API which provides the kind of PoW service to be very useful, and some hacks might be required to mitigate side effects of relentless scraping. These are just hacks, useful today, but the real solution is coming soon. Web3 some people call it, or Cryptocosm is another name of it.
Right now, if I was to ask my users to do a proof of burn for $0.0001 of BTC most of them would just close my site as they don't have any BTC. The process of setting up an account on an exchange, waiting several hours/days for KYC checks to clear, adding a credit card, buying BTC, sending it to a browser extension, and then trying the signup again is a *significant* initial hurdle. If we were in a world where I could assume all my users already owned BTC, that's a different story, but we haven't seen adoption of cryptocurrency anywhere near that level. I don't expect PoW schemes to drive that adoption either, so this seems like a poor solution today.
Do you happen to know if a hCaptcha/mCaptcha-like micro-proof-of-burn tool exists already? I'd be happy to be proven wrong.
That brings us back to options that exist today, which includes every user computing their own PoW. Looking at mCaptcha some more, it uses a SHA256 derivative so it's compute-hard and vulnerable to GPUs/ASICs. The author mentions some of those concerns in an earlier thread [1]. I wonder if a different proof-of-work algorithm would be better, like a memory-hard PoW, proof-of-space, and/or proof-of-wait. I'm skeptical of those too, unfortunately.
In my calculations, with millionth of a cent per transaction, even paying for torrent blocks (64 KB), not torrent pieces (16KB) will soon become profitable.
Unlike blockchain systems, the implementation details of mcaptcha are also totally hidden, and you don't have to maintain compatibility at all.
Someone creates an asic? Great, you can make it not work anymore without affecting any of your users for real.
10+ million down the drain.
I added FriendlyCaptcha to some of my sites, and stopped 100% of abusive traffic. Open source, user friendly, accessible to people with disabilities.
Most of us are not running amazon.com here.
If i may add, in case someone desires a little bit of revenue from a website, one very popular solution is to put advertisements in some places. Well some people consider that a security hole, including me. So i guess the definition of security varies, but the security mania goes on for many decades. I am one of those security maniacs, and any tool to enhance security is important, blockchain is one of them.
I've found that black and white thinking in security is very dangerous, as you often end up with very "secure" controls that have terrible UX, which users bypass completely via byob etc... And pwnage ensues. UX is a primary pillar of security.
We have pretty good guides to what's an expensive computation to everyone. That's how password-hashing algorithms work, much better than BitCoin does. Besides, the existence of specialized methods of compute is hardly determinative. We're not trying to stop TLAs. It just needs to be expensive enough to stop 99%, and at most we'll later update again.
ASIC's take tons of time and money to design.
Unlike blockchain, where the PoW algorithm is part of basic compatibility, it's not at all in mcaptcha - the users see a checkbox. The rest is implementation details.
If someone creates an ASIC, you can break it very easily by changing the PoW algorithm a bit, and no users are affected.
Even if someone has the money to keep up with you, which is remarkably expensive, they will be too slow right now.
That would be hard to change
The genius of bitcoin (not BTC) is that it provides an organized and practical way, for PoW to be used by everyone on the planet. Some people find it strange, because there is an imaginary token created out of pure nothing which represents the PoW. It would work fine, without that imaginary token i.e. bitcoin, just by dollars of euros, but it wouldn't be internet native. The point is always to just send a minimal PoW with every http or tcp/ip request.
Bitcoin is an evolution of that idea applied to digital cash.
IMAP isn't used for sending email. You're thinking of SMTP.
Hardware specialization breaks the economics behind a CAPTCHA. To fight that you need to use a PoW that hasn't been ASIC'd yet, and be willing to change PoW functions at the drop of a hat. PoW functions that stress memory or cache are also helpful here, though you run the risk of browsers flagging you as a cryptominer (which is technically correct, even if economically wrong).
Seems like it should be the same cost (barring the friction of having a wallet, etc.) for both sets of people since Bitcoin is just a commodity and the value is the same to everyone, miner or not.
For example, a miner should value some fraction of a BTC the same way anyone else does, since they can sell or buy it at the same price a normal user can. The fact that they can profitably mine BTC just means they have a profitable business on the side, it doesn’t mean they should prefer to pay for services (or emails) in BTC.
So in order to moderately inconvenience said spammer, you have to make each and every ordinary user wait hours mining a few satoshis' worth of hashes in order to be let in. This is the exact opposite of what you want.
But even if you pick a novel function that doesn't have special-purpose hardware, spammers can still optimize their setups to lower the effective unit cost below what a legitimate user faces, by doing normal miner activities (picking hardware, scaling up, moving to where electricity is cheap, etc.).
Since your goal is to maximally discriminate between legitimate and spam use cases, you'd want spammers to face at least the same per-unit cost as legitimate users.
What's one way you can do that? Well, how about charging actual currency, whether Bitcoin or fiat? Money has the useful property of having the same nominal value for everyone, and not being amenable to further optimization.
In short, forcing users to actually run PoW themselves doesn't really make economic sense. Even if you're avoiding existing hash functions, it's mostly worse than just charging money because spammers have better ability to optimize against it.
And if charging money doesn't work, switching to local PoW is unlikely to be better.
FWIW this thread inspired me to implement Cloudflare Turnstile on one of my pages - highly recommend. As compared to reCAPTCHA your users are never wasting their life away clicking on traffic signs, and as compared to mCaptcha you don’t have to host the server yourself.
As the problem with captchas, is rather who profits from them and who could track users, it seems that with an OCR system to be protected, it is most reasonable to actually give OCR tasks to humans. IMHO this is far more sustainable and people would understand the value: there is nothing bad in improving ML in general. Maybe someone could even define a sensible PoW task for OCR but I doubt it...
I think this whole thing is a big hurdle just because I'm unable to solve visual puzzles. Besides, having a company collecting email addresses of people who are disabled in one way or another and giving them an identifying cookie is a privacy/data disaster waiting to happen.
That being said, I think the audio alternatives for visual CAPTCHAs are also unacceptable. Even if you can hear them, they may be hard to solve especially if they are not provided in your mother tongue. I think we can and should be able to do better by now
If you're going to waste my energy to do proof-of-work anyway, I'd rather you use it for something useful (even mining crypto-currency to pay for server costs) rather than let it go to waste.
eg. in your case this could mean if traffic is above eg. 75QPS then captcha is enabled, and if it's below that it's disabled.
I don't know what tech stack you are using, but nice trick that i figured out was to abuse rate limiting to detect global traffic (doing if branch with rate limit with const as client id)
For a while, we had to just set off pagers when global traffic exceeded a threshold and manually toggle the extra hardening, but eventually it became a lot more reactive.
I guess your tool is asking the registries if domains are registered.
host example.com
If this returns with an IP address, no need to talk to a registrar. Only if there is no IP address, I go with whois example.com host -t soa example.com
which should give you domains that have any DNS record at all, not just an A record.If the registrant creates any DNS records at all, then the SOA will need to exist for the zone to be valid; but I don't recall whether the registrar is or isn't required to publish a zone for a registered domain that otherwise contains no records. (Also whether the registrar is required to inform the registry of authoritative nameservers for the domain at all times from the moment that the domain is registered, and then whether that information would lead to the synthesis of a SOA record.)
I guess I can either try this (if I could find a no-frills-enough registrar that doesn't create any records at all for add-on hosting services or domain parking) or try to take some more ICANN coursework to find out the answer.
Or maybe someone else reading this thread knows whether we can have a domain in practice that is registered but has no published SOA.
This reminds me of the RMT driven botting problem in WoW (World of Warcraft). Instead of fighting the neverending game of cat and mouse against botters, Blizzard just decided to supply the long reprimanded demand for in-game currency by creating the WoW token, and they make money while they're at.
He knew that was the reason for the intentional DDoS due to messages (along the lines of “think it is funny to poison my information do you?”) in the query string of the bulk requests. Like unpleasant fools making a big noise in shops because they aren't served immediately after cutting in line or some such, the entitled can be quite petty when actively taken to task for the inconvenience they cause.
Passive defences are safer in that regard, assuming you don't get into an arms race with the scrapers, though are unfortunately more likely to mildly inconvenience your good users.
Though I did have success once stopping hot-linked images by serving up images that didn't fit well with the sensibilities of the internet forum that was hotlinking them. It had the advantage of looking like the people in the forum posting had deliberately chosen to post the image. Serving different images depending on client ip made for fun too, as they argued with each other about why they posted "that".
Evil. I like it.
If somebody wants to access endpoint you might send him a challenge first. Random text. The client must append to the text some other text chosen by him, so that when you calculate sha256 on concatenated text, first byte or two of it will be zeros.
To access your actual endpoint client needs to send that generated text and you can check it if it results in the required number of zeros. You might demand more zeros in times of heavier load. Each additional bit that you require to be zero increases number of random attempts to find the text by a factor of two.
To make stuff easier for yourself the challenge text instead of being random might be a hash of clients request parameters and some secret salt. Then you don't have to remember the text or any context at all between client requests. You can regenerate it when the client sends second requests with answer.
Honestly I don't know why this isn't a standard option in frameworks for building public facing apis that are expected to be used reasonably.
Generating challenges is really cheap. Calculating a single SHA256 or sth out of a querystring+salt. Can be done on separate layer so requests without valid PoW won't even register on your main system.
I actually do like your puzzle challenge idea from an obfuscation standpoint of making an API less attractive for clients that aren't your own, though. A challenge system + an annoying format like Protobufs instead of JSON is too much work for a lot of abusers.
Biggest problem with DDOS though is that if the volume is even reaching your application server, you're probably hosed.
It means that on this hardware they can make one request per two seconds not thousands per second.
And it affects all of the hardware under the control of an attacker, so his attack becomes thousands times less dangerous.
> Biggest problem with DDOS though is that if the volume is even reaching your application server, you're probably hosed.
That's why I'm suggesting that challenge generation and validation can be done on separate machines. So your application servers can be safe.
Obfuscation is a valid point too.
I thought about it for a bit, made some experiments. Now I think the challenge should be completely random but the server needs to keep track of recently issued challenges and solved challenges to prevent reusing of already calculated solutions. I think bloom filters would be perfect for that because some small percentage of false negatives doesn't matter.
I was keeping an eye on it, because we are hard-capped at 100 QPS to our provider, beyond that and they start dropping our traffic (and it is an outside provider, bundling domain registries like verisign and stuff), which makes regular users break if their traffic gets unlucky.
Anyway, after a week of 40qps, they start spiking to 200+, and we pull the plug on the whole thing: now each request to our endpoint requires a recaptcha token. This is not great (more friction for legit users = more churn) but it is successful. IF they had only kept their QPS low, nobody would have cared. I wanted to send some kind of response code like, "nearing quota".
FTR before people ask: it was quite difficult to stop this particular attack, since it worked like a DDOS, smeared across a _large_ number of ipv4 and ipv6 requesters. 50 QPS just isn't enough quota to do stuff like reactively banning IP numbers if the attacker has millions of IPs available.