News for hackers. Stuff that matters.
Re:Google No Longer Automatically Indexes Websites – WTF? by sph on Monday June 05, 2023 @09:43AM (#36196084)
> On the other side, I've also noticed it appears to be aggressively pruning its index in the past few years, so the fact that it's crawled your site doesn't mean it's necessarily searchable either.

I've noticed this as well. I have a crappy website for my app I need to do better marketing for (not my priority just now), but I've noticed that, for however crap it is, I have received ZERO incoming hits from Google, apart from a couple people that have literally just googled my domain name.

I do not believe for a second there's not a single query done in the 2 months the page has been up, globally, for which my website wasn't a bit relevant. Either that, or the spam problem Google has is much bigger than anyone thinks.

Yet another data point in favour of the Dead Internet theory.

Re:Google No Longer Automatically Indexes Websites – WTF? by gilleain on Monday June 05, 2023 @10:01AM (#36196278)
You could try the google search console - it gives you a view on what hits/clicks have come in over time.

edit: Hah. I notice it suggests using it at the top of the page if you use 'site:..." - and I only get 5 results for my site when the console claims to have indexed 10 times that many!

edit2: Also duckduckgo returns more like 15 hits ...

Re:Google No Longer Automatically Indexes Websites – WTF? by soared on Monday June 05, 2023 @10:23AM (#36196561)
Silly to see people complaining about search results and indexing without backing those claims with data from search console. It’s like devs turn off their brains when it comes to marketing because they don’t like it.
Re:Google No Longer Automatically Indexes Websites – WTF? by sph on Monday June 05, 2023 @11:21AM (#36197513)
Google's bloody Search Console says I got 16 impressions in 2 months for literal searches of my domain name, and nothing else. Funny seeing people thinking I got those figures by reading tea leaves.

Who's the silly one now?

Re:Google No Longer Automatically Indexes Websites – WTF? by PaulHoule on Monday June 05, 2023 @10:18AM (#36196511)
I have all sorts of things that I wrote years ago and I can never find them searching by title unless I put the specific name of the site in the query. I sure can find the slideshare though where some guy from Oracle stole not only my title but much of the content from my blog.
Re:Google No Longer Automatically Indexes Websites – WTF? by rideontime on Monday June 05, 2023 @10:02AM (#36196289)
Dead Internet theory?
Re:Google No Longer Automatically Indexes Websites – WTF? by b800h on Monday June 05, 2023 @10:07AM (#36196360)
From Wikipedia:

"The dead Internet theory is an online conspiracy theory that asserts that the Internet now consists almost entirely of bot activity and automatically generated content, marginalizing human activity. The date given for this "death" is generally around 2016 or 2017."

Not sure I'd call it a conspiracy theory.

Re:Google No Longer Automatically Indexes Websites – WTF? by gspencley on Monday June 05, 2023 @10:26AM (#36196610)
> Not sure I'd call it a conspiracy theory.

What I find funny about that framing is that, regardless of whether or not the theory has merit, a conspiracy theory by definition asserts that there exists two or more people conspiring with the intent to produce the alleged outcome. From what I understand, dead Internet theory alleges no such collusion or intent. I could be wrong but I believe that it merely suggests that the amount of bot-generated activity has come to dwarf human generated content to the point where the Internet is effectively "dead" from the perspective of its original purpose: humans sharing human knowledge.

Re:Google No Longer Automatically Indexes Websites – WTF? by PaulHoule on Monday June 05, 2023 @10:39AM (#36196780)
10 or so years ago I wound up blocking everyone other than Google in my robots.txt because I was sick and tired of webcrawlers from China crawling my site twice a day and never sending me a single referrer. Same with Bing. Back when I was involved with SEO the joke was you could rank #1 of Viagra on Bing and get three hits a month.
Re:Google No Longer Automatically Indexes Websites – WTF? by roncesvalles on Monday June 05, 2023 @10:38AM (#36196764)
The "conspiracy" is that companies like Google are pretending like it isn't dead.
Re:Google No Longer Automatically Indexes Websites – WTF? by tankerkiller on Monday June 05, 2023 @10:38AM (#36196769)
At least so far according to Cloudflare bots consist of around 1/4 of all internet traffic. But that could be pretty far off depending on how they get those estimates.
Re:Google No Longer Automatically Indexes Websites – WTF? by esperent on Monday June 05, 2023 @11:17AM (#36197441)
The figure I saw most recently was 42%. Weirdly my brain can remember the number but not where I saw it.

But what I'm curious about, whichever number is true, is whether people mean "malicious bots" when they say this, or just any kind of autonomous agent. And also whether they are counting volume of data or simply network requests.

Because if by "bot" they just mean "autonomous agent making a network request" then honestly I'm surprised the number isn't higher, and I don't think there's anything wrong with it. Every search crawler, every service detector, all the financial bots, every smart device (which is now every device) and a thousand other more or less legitimate uses.

Re:Google No Longer Automatically Indexes Websites – WTF? by rchaud on Monday June 05, 2023 @11:49AM (#36198049)
This very link had a Cloudflare "prove you're human" screen that prevented me from reading it.
Re:Google No Longer Automatically Indexes Websites – WTF? by el_snark on Monday June 05, 2023 @12:06PM (#36198378)
I've got a script for parsing my web logs which removes all the lines which match persistent indexers/bots/scrapers and any obvious automatons. Logs generally shrink to 40-50% of their volume, so I'd at least double CF's estimate.
Re:Google No Longer Automatically Indexes Websites – WTF? by pixl97 on Monday June 05, 2023 @12:58PM (#36199276)
https://www.youtube.com/watch?v=kL8rHf_idt0

Thoughty2: The internet has died

In this video they rename it from 'theory' to 'prophecy'. As in the internet isn't quite dead yet, but its rot filled bloated body is near its dying breath.

Re:Google No Longer Automatically Indexes Websites – WTF? by wahnfrieden on Monday June 05, 2023 @10:14AM (#36196467)
use intext:
Re:Google No Longer Automatically Indexes Websites – WTF? by mattgreenrocks on Monday June 05, 2023 @11:58AM (#36198247)
Same issue for me. Message is clear: be relevant to be indexed. And everything indexed is relevant.