Companies spend thousands of dollars on these anti-bot solutions and then they are so misconfigured that using a specific user agent or faking browsing via mobile, bypasses them. Real life stories.

Often this is because you are hamstrung by old mobile apps or TV apps that can't be updated forcibly and so you break users. So your making a trade-off of user pain and bot deflection. So many times this is actually known and on purpose. Botters hitting that loophole helps prioritize closing that loophole in an agile customer experience and makes it easier for engineering and product to prioritize. Real life stories

Just shared some of these stories here https://twitter.com/pigivinciguerra/status/15715437943480893...

Totally agree.

Just want to say THANK YOU for this insight. It never occurred to me, and I just checked out one of the big sites that I scrape for a side project and, lo and behold, you are 100% correct. Found their origin in Censys in about 30 seconds and I've never been able to crunch through their pages more easily.

To others out there who explore this: As with all scraping, be gentle! If you start pounding on someone's origin server directly, you're much more likely to be noticed than if you're pounding on something behind a CloudFlare cache. Set rate limits, scrape during off-peak hours, etc. Be a good scraping citizen.

How did you do it? Asking for a friend who would like to get to the webservers behind CDNs that front Russian websites.

> or if https://developers.cloudflare.com/ssl/origin-configuration/o... is used.

How is using CF’s origin CA preventing the connection to the real backend in order to bypass Cloudflare? you cam just ignore the SSL error couldn’t you?

They probably meant to link https://developers.cloudflare.com/ssl/origin-configuration/a... where Cloudflare uses a client TLS certificate to pull from the origin and the origin should be configured to reject requests without a client certificate.

In addition, I think CF provides a list of IPs to whitelist to only allow access from their servers.

One more technique to find the backend IP address of the web-server is to lookup the DNS record history of the domain in question. Usually, the admins will test their website with with SSL on a real domain to ensure it works.

Only afterward, they will protect it with Cloudflare/Akamai/Cloudfront, etc, which means you'll have a DNS history trail to lookup.

I've had good luck with services such as SecurityTrails, Virustotal and CompleteDNS for DNS history.

Once you find the original A record IP address, you can put it in your hosts file, pointing to the domain. Then, you will be able to access the site, bypassing the CDN without issue.

This is what I expected the article to be about. I would wager a lot of shops don't to the whitelisting. If they wanted to be really intense they could do authenticated origin pulls.

AWS CloudFront with S3 recommends that you just set your S3 to require a specific 'Referer' header variable and you set CloudFront to send that custom 'Referer' with each origin request.

Seems to work great when you use something like a GUID, and no need for IP whitelisting.