Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving up some JavaScript to check my browser.

The information I'm trying to get is public and can be accessed without any kind of authentication. I'm willing to bet that they flipped the anti-bot stuff on their load balancer on for the entire site instead of doing the extra work to only enable it for just electricitycompany.com/myaccount/ (where you do have to log in).

I also asked the company if they'd be willing/able to push the power rates out via the smart meters so that my interface box (Eagle-200) could pick it up, they said they have no plans to do so.

The next step is to scrape the web site for the provincial power regulator, which shows the power rates for each provider. Of course, the regulator's site has different issues (rounding, in particular), I haven't dug any further to see if I can make use of this.

All of this effort to get public information in an automated fashion.

Social media platforms has made physical appearance as the first class citizen of the reputation economy. I'm not even talking about those platforms which outright bury content from the disabled as a policy, I'm talking about those platforms whose algorithms favor selfies, videos over text/URLs and thereby putting those with accessibility issues in severe disadvantage.

Why would you use such platforms one might say, Do something which has nothing to do with the reputation economy they might add; Well have you looked at LinkedIn lately? LinkedIn has become ubiquitous with professional job search and 30 second video intro is the very first thing on the profile, not the skills which the platform was meant to be when it was launched. One must be naive to claim that the physical appearance on that video or profile picture doesn't affect the job prospects(Several studies have stated otherwise).

It's not just the physical appearance, The action of creating videos or posting photos itself is hard as a time-constrained person[1] and so I think it's reasonable to ask the platform to allow bots to post deep-fake videos of the user doing silly things which these platform expects from an average user.

[1] https://abishekmuthian.com/time-constrained-person/

Fortunately there are plenty of tools to handle this, and at a hobby level not particularly resource intensive. Something like this is simple and reliable in many cases: https://github.com/berstend/puppeteer-extra/tree/master/pack...

Not really. In a lot of cases websites use JavaScript to call some API along with some on the fly generated token to prevent abuse.

As long as that token isn't captcha you can reverse engineer the site to do scraping without javascript and that is so much faster than browser based scraping.

Its just a cat and mouse game. After few year I think hardware attention etc will come to play which can mitigate bot issue somewhat.

A product protecting against credential stuffing might as well prevent denial-of-service as well.

Primarily it is for automating web applications for testing purposes

Here was I thinking it was just a tool to scrape all the js I am too lazy to reverse engineer.

Also see: https://www.ft.com/content/0e746280-e72c-4087-9c0d-df2a7af82...

Feb 2021 FT: "Bills mount in Texas power market after freeze sends prices soaring: Financial casualties emerge as grid operator Ercot requires billions in payments "

Thanks to deregulation, the electricity rate isn't the only thing you pay for though. There is also a Transmission Charge, Distribution Charge, and Local Access Fee. These are all per-kWh charges and change very rarely.

In October, my electricity rate is $0.10730/kWh, but my total cost is actually $0.16346/kWh plus the per-day charge ($0.202/day). Tomorrow the November rate will be published.

[0] https://ucahelps.alberta.ca/cost-comparison-tool-result.aspx...

It seems to me from a life-safety angle that their energy would likely be far better spent on recommending smoke alarms, CO meters, and periodic cleaning of dryer vents than on recommendations against sleeping with washing/drying machines running.

In short - if there's a fire it's much better that you're awake and up already.

https://www.flickelectric.co.nz/pricing-and-plans

https://www.pauatothepeople.co.nz/cheap-as.html

Aside from this, peak / off-peak pricing is not unusual.

- committing clickfraud to game ad and referral revenue systems

- posting fake or spam reviews and comments

- generating fake behavioral signals to help bypass CAPTCHAs to help create accounts on other sites that can post spam comments

- validating stolen credit card details

- screwing with your metrics collection if you can't identify them as bots

All of that is enough reason for sites to use bot detection and blocking technology. The fact that the same tech also has some utility against accidental or malicious traffic-based DoS is also a bonus.

To be clear, "validating" is an industry euphemism for stealing, just for a different purpose. How do you validate the card is live? Run a real transaction through it and mark it based on the result. But what do you run for this real transaction? Well, whatever you want. Typically it'll be something to avoid suspicion as much as possible, but the thief gets to pick what they test it with, so why not pick something that they'll personally benefit from? There are so many online games with purchasable currency these days, it's hard to choose.

They won't cash out on them or buy items. Instead, they'll collect cards from a source (skimming, hacking, whatever), validate them by adding them to a website that does an authorization (those $1 checks that never get committed). They can then sell them wholesale for a premium compared to non-verified cards.

I've been away from that world for a while but remember that more serious operations will separate the cashing out part (either money or goods) from their acquiring / validating operation because the former carries more risk.

There's also an interesting episode of the darknet diaries podcast (https://darknetdiaries.com/episode/85/) about card cloning which I found interesting.

Not necessarily. Some online services, particularly shops that do home delivery, may give their customers the possibility of adding a card to their wallet and perform a verification as part of the process. As a result, it becomes possible to validate a stolen card number without performing an actual transaction.

The most amusing to me are the ones from “Microsoft” to alert me that they have detected malware on my computer.

And yet if you dare browse the web with TOR or a VPN or sometimes just happen to be on a small ISP[0][1] then you're being punished immediately. You solve your cloudflare-supplied captcha because you may be a bot (you're not, and the dangerous bots will not be defeated by this anyway, but some humans will be), and then you get an error from the website itself because it runs a secondary bot detection thing. And you weren't even anywhere near anything "dangerous" like a checkout page.

[0] My parents use a regional small ISP (but locally very popular) that serves around 50K customers. My parents also use a regional bank (a Volksbank, and those are members of a national association that provides all kinds of services). Suddenly that bank would not even let them see the bank's front page. After some back and forth on the phone support line it turned out the bank had recently deployed some "advanced" bot detection, one that had a whitelist of residential-ISP-associated AS/IP ranges, and of course whoever compiled and maintained that list had forgot to include that small local ISP. For that regional bank it meant they had just shut out a very significant number of their customers (and potential customers just trying to look up what the bank offers), as there was very likely a huge overlap of people using that regional ISP and that regional bank (both are regional, after all). It also was something they couldn't fix themselves, as the "online banking" stuff was not in-house but was run by the national association (which probably used some bot-detection as a service provider). It took the bank (or rather, the national association) a few weeks to fix. Mind you, the last few years that bank has been heavily marketing a cheaper "online only" account, only online banking, online support, and access to the self-serve ATMs and banking terminals, but no face-to-face or even ear-to-ear human interaction. Try contacting "online support" about "the website outright refuses me" when the "online support" is only available on that website. Kudos is you're smart enough to switch to their mobile app, as your phone uses a different ISP, unless you forget to turn off wifi. That's the advice my parents got from the phone support (they sill have an account type that not online-only).

[1] When I recently visited my parents, from the wifi [same ISP as in 0] I couldn't open the website of a bakery too look up if and when they would be open on a Sunday. Some error message about "this website is not available in your network" (English text, for a German bakery... suspicious :P). I could open it via my mobile, tho. I could open it from my regular ISP when back home (in another city) again. Mind you, that website is purely informational and has no "interactive" features let alone let's you buy anything. It's just static text and some pictures.

Smaller ISPs are also more likely to have issues with CPE getting compromised and routers running botnets within the comfort of your home. There are services which invite people to sell their residential bandwidth in return for money, this can potentially have a disproportionate impact at smaller sample sizes.

Networks can declare themselves to be ISPs. You could check if your ISP shows up as an ISP in peeringDB.

https://networkengineering.stackexchange.com/questions/44585...

They are reportedly very proactive when it comes to CPE security, as well, up to giving customers a proactive phone call when they see somebody is using equipment with known vulnerabilities (customers are allowed to operate their own equipment as long as it is deemed compatible, most will use remotely managed equipment, tho, I believe; my dad used to use his own DSL router and once got such a call if I remember correctly. He switched over to their fiber now and managed equipment).

Their AS is indeed identified as an "ISP" in peeringDB.

While I would be extremely surprised if the company was doing shady things, you surely got a point that a small ISP like that could suffer more in reputation from some few customers being up to shady things, including sub-lending the line. I am pretty sure that is against the ToS, but enforcement is a problem of course. Especially detecting such traffic without violating German privacy laws is probably a difficult task, but it's not impossible.

Stuff like this is a pain beyond pain. I really hope that the clients you mention know that they piss off a proportion of their users with every move they take.

With all due respect, if the tech can make a large impact on the problems mentioned above, I'm sure it's an easy decision for the big companies to take decimating bot activity over the tiny minority of users who proactively decide to disable JavaScript.

Said as someone who uses NoScript, FWIW.

I'm a linux user myself, so I know for a fact that neither my previous employer, nor other bot vendors, will block linux user agents in particular. Customers generally don't mind a universal requirement for JS execution, so that's just a fact of life. We generally did try to avoid blocking privacy focused browsers, though. We certainly monitored false positive rates and knew pretty well how we affected users.

Have you tried not using these things? Anonymity is exactly what bots want. They want to be able to post a spam message every single second and be impossible to ban since they are anonymous. The internet can't function if people are allowed to be anonymous.

You must have missed the first 20 or so years of its existence, if that's your position.

Stopping abuse has always been a game of trying to deanonymize users in order to try and ban the harmful ones.

All of these have independently caused me to get into endless ReCaptcha loops: firefox on android, smartphone with unusual screen resolution, clean browser profile with VPN.

It's so common that I now default to using duckduckgo, which never blocks me. I doubt DDG has a lower DDoS/Resources ratio than Google. Some companies are just lazier and less principled than others.

"Unusual" = not Chrome and doesn't allow tracking scripts.

Switch to Safari with an ad blocker for a week, see how many more ReCaptcha prompts you get.

Fortunately, almost all of the websites I visit with my anonymized browser aren't places that I wish to attempt to post a message. Unfortunately, I can easily run into defenses of an entire site when the problem is spam sending.

Parent poster trusts his bank, and his bank would trust his once it knows he's not an fraudster, so maybe it's in everyone's interest to just allow the javascript for that one site.

Nobody wants to spend time trying to stop these bots. It is, however, a very necessary thing to do.

I’ve noticed most sites won’t let you search business fares efficiently, so I made my own for Google Flights which only worked for like 6months until they added bunch of changes that made it near impossible to scrape.

Well, because protection is not a binary thing - either being 100% safe or 100% not working - instead it's a proportion between the skill/effort/time needed to break in, and the reward you get for it.

To stop majority of attacks you don't have to be absolutely unbreakable, you just need to make it hard enough for majority of attackers so that it doesn't payout for them compared to the value of the data you're protecting. And that's where anti-bot SW has it's place, it slows done spiders and global attacks, forcing for custom tailored scraping that is constantly being fine-tuned, infrastructure to hide your IPs, and that makes the operation way more expensive and harder to run continuously...

Now the best part... one division (big team) of our company worked for the (national carrier) airline , one division of our company worked for the resellers (we had a single grad allocated to web scraping). The airline threw ridiculous dollars at trying to stop it, and we just used a caffeine fueled nerd to keep it running. It wasn't all fun though, they'd often release their new anti scraping stuff on a Friday afternoon. They were less than impressed when they learnt who the 'enemy' was. Good times!

Can you say more about this? What do you mean by "Once you get to selenium it's usually over", and how do you manage cold starts in Selenium and emulating heavy usage?

Say your program starts right now, I assume you don't go through "adding heavy usage" to "warm-up", then get down to business, correct?

Travel information is also one of those services where it’s not weird for a significant number of their users to use it quite heavily, making behavioral detection more difficult.

Together, there's little to detect different than a regular user. The reason why the residential IPs is given heavy importance though is that it's the one part that costs a lot of money if you need enough of them you need to use a proxy service and you transfer a lot of data. Entry level pricing is over $15/GB for high quality services.

This sketchy company lets mobile app developers monetize user base by letting other people pay $$ to route requests through random people’s mobile IPs: https://brightdata.com/

Similarly NordVPN owns Oxylabs (who mostly hack routers and cameras and sell those as residential IP’s).

Mostly to avoid hitting 'ddos protections' or other security bullshit that doesn't really make sense on 1 daily request or so.

I think these companies used to go for extension developer now it seems they have found new idea to implant malware on apps which is not easy to detect.

As a scraper operator on a mobile data connection all you need is a new useragent and browser fingerprint, there's no easy way for a scraper-blocker-operator to tell that you're not a totally new person.

There's various captcha solving services where you pay in bulk per captcha and submit data via an api.

It's like writing a game bot with Java robots and pixel detection. It may is inefficient, may takes longer to make than a network solution. But I have yet to be detected anywhere

I disagree. Obviously there is no way to 100% stop scraping, but a for a rather small amount of $ you can implement some measures that make it harder. Services like https://focsec.com/ offer ways to detect web scrapers using proxy/VPNs (one of the most common techniques) for little money.

> Nobody pays those vendors $10m/year to frustrate web crawler enthusiasts, they do it to stop credential stuffing.

Keep in mind that they may be legally or contractually forced to do this. Think of Netflix who are investing heavily into their Anti-VPN capabilities, most likely because they have contracts with content publishers & studios that force them to do so.

What do you mean?

Of course if everyone did this, then all of your logins would have the same password (your email login).

There is vast amount of profit available in doing just that (see e.g. GOOG and FB market cap). Even companies that truly have no intention of exploiting data collected as a side-effect of whatever product line they do, nearly always eventually end up going for that profit line. Because passing up on more income merely due to moral considerations is too much of a temptation for a company to be able to resist in the long term.

How is there not an equilibrium here that cuts off credential stuffers? I'd naively imagine the residential IP providers have some measure of bad actors they themselves use to determine if a client is worth it, and that someone getting all your IPs blacklisted would get dropped pretty quickly.

I don't know about $10m/year, but many sites block bots just because they don't want competitors to access publicly available data. Which is bullshit.

I know how bad this issue is, and I wouldn't take down this repository. Anti-bot software does not work, anyone who pays 10m per year to have it simply has too much money.

2FA, especially app based, has been proven to work really really well.

2FA is a good security feature but it does not help against web scraping. Credential stuffing and other 3rd party attacks? Yes, it _can_ help. But it does not always help. There's a phishing group that has seemingly specialised on getting people to click the green confirm button in their Duo app... ¯\_(ツ)_/¯

Check https://github.com/revalo/duo-bypass for a python script that can be used to automate Duo tokens... Has some code from me. There are similar scripts for all the other well known OTP Apps...

At the bare minimum, TFA stops most attacks. That's a whole lot better than the current situation.

It seems that the Duo core app is a variant of HOTP?

What's the name of the phishing group and any details on them? There was a Defcon or Black Hat video where they would constantly send a push approval to the mobile which was not PIN protected and most people would click on it. Don't remember which OTP generator it was.

100% reliance on a phone which is easily lost, broken, stolen, etc. without backup is really bad IMO. My bank (Revolut) only had a mobile app, and no way to contact them outside of it (I tried...) I need to switch banks.

1. https://blog.revolut.com/introducing-the-revolut-web-app/