How are we "stopping search innovation"?

Thanks for taking the time to reply.

You mentioned about "legit" crawlers, what defines a "legit" crawler in the eyes of Cloudflare, and what happens when Cloudflare suddenly decides it does not want to honour that "agreement"? What happens if/when Cloudflare is sold, or the contact who greenlit these smaller "legit" crawlers moves on and decides that it no longer agrees with said website anymore?

Is a price comparison site a "legit" crawler? What defines a "bot" vs a "crawler" in the eyes of Cloudflare?

Would you need to notify your customers that you now also allow additional crawlers access to their sites, or would they need to opt into it via the Cloudflare dashboard? What happens when you have a falling out with my said company (it happens, relationships sour) and suddenly we can't make contact, then suddenly customers websites aren't being scraped because we're bots?

If you are a price comparison site getting blocked by cloudflare, site owners may be losing sales, and that's good feedback.

Depending on the industry, etc..

I really can't think of a good solution. But that's the tricky position Cloudflare is in - how does it balance everything.

How can you expect someone to crawl a bunch of websites if they are actively blocked from accessing it? Now, you might say users can whitelist bots in their robots.txt file but then again will the person creating the engine individually ask companies to allow them to crawl?

Also, slightly unrelated but Cloudflare protected websites are almost impossible to access via tor, the captcha never succeeds.

Yes, I've never understood why it's seemingly so important to CAPTCHA me before serving me less than 100kb of read only plain jane HTML. What sort of "attack" is this stopping? I'm pretty sure the CAPTCHA itself is bigger than half the sites it blocks me from reading.

When I had a site that had millions of pages, I found that sites like Baidu would crawl my site as often, if not more often than Google.

I already felt the relationship with Google was parasitic, but I looked through my logs and never found a single hit that came from Baidu and many of the other search engines that would overload my site.

I was looking at a substantial part of the site running costs going to supporting web crawlers that were not doing anything (1) to help me, or (2) to help end users (if they don't want to send Chinese users to an English-speaking web site, why crawl the site?)

So like it or not I am inclined to only allow Google and Bing in the robots.txt because Google is the only site that sends a significant amount of traffic and because Bing sends some, and Google needs some competition.

There are web crawler behaviors that are annoying: harvesting email addresses, overloading your site, etc. But how do you know who is doing something wrong with the data and who is just collecting it do do nothing with it? (Probably 95% of web crawling ex. Google.)

This sounds like you're onto a reasonable “legit” factor: does the crawler honor robots.txt? Baidu would be legit because they don't lie about their identity and if you put a rule in your robots.txt file they'll honor it.

And what about crawlers with even more limited scopes (targeting only a handful of sites) that they can’t possibly be called search engines? Are they ever considered legit?

They're only a gatekeeper because sites voluntarily enter into commercial agreements with them. There's no coercion or manipulation like Google AMP.

No such filters should exist. Is it really that awfully bad without anything but basic filters (ban an IP for flooding)? Are there like, operations that try and spam every single Cloudflare-hosted website 24/7?

Legitimately curious if your anti-bot measures come from actual bad experience with the internet or is it just a liability limitation move? (Namely to reduce potential suing surface by angry data owners and/or three-letter agencies.)

---

Basically, if I am experimenting with a basic crawling program and I hit websites A and B 20 times each in a space of one hour, is that really deserving of a captcha or extra auth methods?

Not flaming but I am really curious. Do you have any data and rationale posted somewhere that go into deeper detail about why Cloudflare's bot detection is how it is?

That isn't to say that the customers are savvy and have a good understanding of different types of automated traffic and which automated traffic is harmful and which is benign. Many have a quite naive understanding that doesn't extend beyond "bots = bad, unless it's Google" and dial protection settings to the max for no good reason.

Recently your company announced partnership with Internet Archive, but if CloudFlare want to continue play a role as behevolent party everyone should have equal access to this data. Yeah it means that some bad actors will be able to easily scrap the web too, but...

CloudFlare service can't prevent scrapping anyway. There are shady residential proxy networks, services to bypass captcha and scrapping software like Zennoposter. It's possible to make scrapping more expensive, but bad actors don't care because they have money. Unfortunately enthusiasts, open source projects and small companies don't have enough resources to do the same.

I think you might not understand that it's site owners like me who want to stop scraping. It usually comes from specific bad incidents, like copycat sites stealing our content and work.

Cloudflare wouldn't block scraping if website owners didn't want it. And website owners can easily disable this protection.

Also I'm talking from experience because I been on both sides of the fence: doing scrapping and implementing protection. So yeah your CloudFlare protection will deter 10% of bad actors, but will also cut off 99% of enthusiast / research efforts or users of niche software or browsers. Still anyone with $1000+ budget will scrap whatever they want.

[1] https://en.m.wikipedia.org/wiki/Sealioning

Or in practice - each node quickly runs into a CloudFlare captcha preventing it from indexing content for a few hours/days. Since CF fronts a lot of the useful internet these days, it means it's effectively working against distributed indexing with its current captcha solution.

Which makes "contact us for an exception" a no-go, as the relevant source IPs will constantly be changing.

In fact it's a textbook definition of a non-scaling solution.

That said, I don't think it's a good situation that this is the solution rather than a proper, documented position that people can work to.

If this doesn't at-least meet your definition of "reach a human at Cloudflare", I'm not sure what will.

Individual companies having so much power gives me the willies.

But let's suppose someone is building a new cool search engine and our ML stuff is blocking them. Then... contact us/me.

(but Google is allowed because Google was first to market)

Yes, you must adhere to the controls that site administrators put in place, like Cloudflare.... You don't get to blast my site with requests, just because you want to...

(b) But you’re a-ok with Google doing this. Gated communities aren’t really good for anybody but I see what you are saying.

The same is true online. Apple's walled garden has kept hundreds of millions of people safe on their device. It's why iOS malware isn't a thing.

> Cloudflare stops much more than just blasts

Exactly. There's even more benefit to Cloudflare than just DDoS. Captcha's for stopping credential stuffing, for example.

But hey if I pay Cloudflare enough, then I’ll get to blast your site and possibly stuff creds at the same time :/

[1] - https://en.wikipedia.org/wiki/Robots_exclusion_standard#Craw...

So no problem, time to move on web search is no longer exciting

1. Filtering ever increasing trillions of spam/clickbait pages

2. Figuring out which results are useful information vs corporates trying to sell something.

Those problems are not solveable by a couple guys in a garage

Reminder: Google Is An Ad Company. Are you sure they actually want to solve search? Their primary interest is to be that corporation selling you something.

It turns out that he was right, sort of. The vanilla attach server into the internet, server-to-client IP-network is pretty much dead. It has been replaced with CDN's , private delivery networks, cache on top of cache. Cloudfare, Amazon, Google and MS are the connection points for the IP. Their internal network infrastructure transfer most of the data.

The fact that 98% of traffic goes trough this new infrastructure, allows some still plug their server to the net raw and their traffic still gets trough.

Terrestrial optical networks operate far, far below capacity to create artificial scarcity, which is to a certain extent necessary to recoup the capital expenditures in a competitive market, and is to a certain extent an abuse of an under-regulated natural monopoly.

If you could eliminate all adversarial factors and put every data service subscriber's payment for a single month into a pool, and that pool purchased only transceivers, passive optics and switches, and this hardware was distributed to every network operator perfectly fairly based on its contribution to the global maximization of available network capacity, then the delivered capacity to the end user could increase by something like 4.5 orders of magnitude with no substantial change in topology or subscriber or provider costs afterwards using the existing fiber, with a few more orders of magnitude possible with a fatter tree before the backbone costs explode.

With DWDM you can carry 100+ channels of 100Gbps over a single fiber today, with commodity, off the shelf components. Most fiber in the ground today is probably still lit with a single wave of 10G, if it's not just dark.

This distribution model is not even remotely a technical necessity, it's an arbitrary local minima reached largely by exploitative market distortions and adversarial economics.

Doesn't mean that 98% of the value of the Internet results from this traffic.

Even if you discount all of the web, there still lots of applications using the federated model (e.g. SMTP) or peer-to-peer (e.g. Crypto, VOIP), that require end-to-end connectivity.

My original point is about capacity and if the old internet could work today. It seems like I'm correct, but I'm not so sure. I would like to see other opinions.

Every response so far is "there exists". The real issue is if the internet could do everything without caching data near the edge.

It's just like Windows - just because 95% of the Internet does something one way doesn't mean it doesn't suck, isn't more complicated than it needs to be, and doesn't cost more than it needs to cost.

Anyone with a little bit of bandwidth and a Raspberry Pi can run a web server, even with dynamic content.