News for hackers. Stuff that matters.

A comment on Google No Longer Automatically Indexes Websites – WTF?.

Re:Google No Longer Automatically Indexes Websites – WTF? by tyingq on Monday June 05, 2023 @09:20AM (#36195817)
It seems obvious to me the importance of organic results has fallen over the years. There's plenty of queries where you have to scroll down a fair amount to even see them. Pushed down in favor of ads, various widgets, content sourced from Wikipedia and published on google's urls, etc. Things that sit below the fold get less overall money, time, and attention.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by hackernewds on Monday June 05, 2023 @09:29AM (#36195919)
With generative AI search results, soon you won't even be able to know whether your site was used for the result or not. Lots of no click queries, resulting in no traffic for the publisher
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by userbinator on Monday June 05, 2023 @09:33AM (#36195971)
The much more worrying fact about AI is you won't even be able to know whether the information you're getting is true. I always scroll past the crap at the top to get to the actual site results.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by danielheath on Monday June 05, 2023 @09:44AM (#36196104)
I haven’t been able to estimate that with confidence in a decade; SEO blogspam has seen to that.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by nicbou on Monday June 05, 2023 @10:17AM (#36196497)
The content is true to the extent that a copywriter paraphrased another piece of content that is more authoritative.

My work is getting paraphrased a lot, and they usually get the gist right, although they have zero clue of what they're talking about.

[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by Karunamon on Monday June 05, 2023 @09:46AM (#36196131)
Given the web as it is today is infested with clickbait, "native content", clout-chasing, undisclosed sponsorship, and other such pathologies, I'm not convinced that the fear of AI making truth more rare is rational.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by phpisthebest on Monday June 05, 2023 @11:18AM (#36197457)
They are the same problem, what do people think LLM is trained on.

"clickbait, "native content", clout-chasing, undisclosed sponsorship, and other such pathologies" is exactly the data set they used to train ChatGPT

[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by Karunamon on Monday June 05, 2023 @11:21AM (#36197515)
Not once have I received a response from ChatGPT that meets any of those definitions.

…Perhaps the robots are better at truth than you are giving them credit for? :-)

[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by Pxtl on Monday June 05, 2023 @12:16PM (#36198512)
I mean, a matter of degrees still matters. The difference between a bad cut and a sucking chest wound.

What happens when somebody points the chatbots at Reddit? Wikipedia?

[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by Karunamon on Monday June 05, 2023 @07:08PM (#36205545)
What makes you think they haven't? There was already upheaval at Stack* as a result of people writing trash answers with GPT help.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by randomdata on Monday June 05, 2023 @09:48AM (#36196157)
That might have been worrying if “I” was known to reliably provide true information, but it never has, so we’re used to knowing that information probably isn’t. Adding an “A” to the equation changes nothing.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by Retric on Monday June 05, 2023 @10:16AM (#36196492)
LLM’s are significantly less likely to be accurate but are surprisingly good at fooling people. The problem is our existing BS detectors no longer work well. It’s surprisingly close to talking to a talented con man.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by watermelon0 on Monday June 05, 2023 @10:26AM (#36196604)
It was never completely reliable, but the situation was a lot better in the past, when SEO spam was not so prominent.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by randomdata on Tuesday June 06, 2023 @01:54AM (#36208689)
Was it? Go down to the local coffeeshop for the daily gossip and you'll hear all kinds of things that aren't true. People love to make stuff up and we have always needed to rely on the concept of credible sources. There is nothing to suggest those are going away.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by gs17 on Monday June 05, 2023 @10:16AM (#36196490)
Yeah, had this happen already. I remembered some normally purely carnivorous type of animal had a herbivore species. It was spiders, but I searched for snakes first. One of the top results was an article about how boas can be fed a diet of fruit (they cannot), which must have been AI written with how many other semi-nonsense articles that site had.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by user_666 on Monday June 05, 2023 @01:52PM (#36200165)
This kind of problem is also showing up on Quora. Some of the answers I've spotted are so obviously wrong that you can tell a person didn't write them.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by baxuz on Monday June 05, 2023 @11:28AM (#36197655)
As soon as I see a url that has the exact search query I typed in, I know that this is a content-farm site at best, and a LLM generated one at worst.

And these have started to become > 50% of my search results.

For the query "discord overlay not working wow", I get the following as THE SECOND RESULT: https://freeholidaywifi.com/discord-overlay-not-working-wow/

[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by xkcd1963 on Monday June 05, 2023 @02:14PM (#36200543)
I mean, what is written on the internet you shouldn't take for granted either
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by lazypenguin on Monday June 05, 2023 @09:46PM (#36207034)
I asked ChatGPT to give me links to some sources for one of its answers and it responded it didn't have access to the internet. I think this could be "solveable" by adding a "show your work" or "provide references" kind of feature in a future iteration.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by ulfw on Monday June 05, 2023 @09:29AM (#36195923)
And let's not forget half the 'organic results' are either links to quora (for text) or pinterest (for images). Both largely behind logins.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by slig on Monday June 05, 2023 @09:38AM (#36196017)
Must be really nice being friends with Google insiders.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by tyingq on Monday June 05, 2023 @09:44AM (#36196096)
They even describe how to mark up your paywalled stuff to help them differentiate it from cloaking.[1]

"This structured data helps Google differentiate paywalled content from the practice of cloaking, which violates spam policies."

Which is just odd to me. Why present a paywalled search result, when the market is so fragmented that the odds the user has a subscription are so small?

[1] https://developers.google.com/search/docs/appearance/structu...

[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by thatguy0900 on Monday June 05, 2023 @09:57AM (#36196239)
Feels like a no win situation for Google. Do you show results that people don't want to pay for, or do you not show any subscription needed results and people accuse you of monopoly behavior only showing sites running adsense.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by tyingq on Monday June 05, 2023 @10:01AM (#36196284)
Maybe a middle ground with a button for "I don't have a subscription for this site, don't show non-free content from it to me again".
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by q87b on Monday June 05, 2023 @10:14AM (#36196460)
One should not need to use an account for Google to basically work.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by emaginniss on Monday June 05, 2023 @11:34AM (#36197782)
ChatGPT requires an account to work at all
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by account42 on Tuesday June 06, 2023 @09:20AM (#36212254)
So since there are serial killers it's OK for me to stab you as long as I stop there, right?
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by q87b on Tuesday June 06, 2023 @09:23AM (#36212293)
I don't see the context here but if you want: guess why I never used it.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by hedora on Monday June 05, 2023 @11:36AM (#36197820)
DDG allows to to set toggles like this without logging in.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by CamperBob2 on Tuesday June 06, 2023 @01:16PM (#36216043)
I for one am willing to meet them halfway on that. Trouble is, they won't even give me 5% of the obvious options I'd find valuable as a user.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by BarryMilo on Monday June 05, 2023 @10:23AM (#36196567)
hahah, giving the user control! Funny stuff.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by morkalork on Monday June 05, 2023 @11:28AM (#36197654)
Which isn't such a foreign concept for them, there's a "no toll roads" option for driving directions on google maps. Why can't search be the same?
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by Pxtl on Monday June 05, 2023 @11:37AM (#36197847)
Right? It doesn't have to fully elide the result even. If the top of the SERP said "3 results hidden because they are paid sites on your non-subscribed list" I would appreciate the info.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by caoilte on Monday June 05, 2023 @12:23PM (#36198652)
They used to have an option to permanently block sites from your google results. They quietly dropped it even before they dropped reader.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by account42 on Tuesday June 06, 2023 @09:17AM (#36212202)
They had a simple rule that they index what users can see and if you try to cheat your way around that you get ranked down. Sticking to that for everyone would have been an iron clad defense against monopoly accusations. Instead they chose to help certain corporations (or at least corporations with certain business models) present different content to the bot and users.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by chongli on Monday June 05, 2023 @10:48AM (#36196918)
"This structured data helps Google differentiate paywalled content from the practice of cloaking, which violates spam policies."

Which is just nonsense. A paywalled site showing different content to GoogleBot than to ordinary users is cloaking.

[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by account42 on Tuesday June 06, 2023 @09:14AM (#36212171)
There was a time that Google tried to prevent SEO. Now Google hosts and attends SEO conferences.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by tremon on Monday June 05, 2023 @11:40AM (#36197888)
Add Steam forums to those. 9 out of ten times, game walkthroughs or hints can only be found on Steam, according to the search engine. But only if you have a login there, of course...
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by user_666 on Monday June 05, 2023 @01:53PM (#36200182)
Since when did Steam forums require a login?
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by rightbyte on Tuesday June 06, 2023 @02:00AM (#36208733)
I don't think you need to login to Steam forums? Edit: Just tried. You don't need a login.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by tremon on Wednesday June 07, 2023 @05:01AM (#36224523)
Indeed, I just checked, it's not the entire Steam community site. Looks like it's a per-subcommunity (per-game?) setting whether the community section is open to the public or not. Luckily there's a lot of content that's not behind a login wall.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by account42 on Tuesday June 06, 2023 @09:23AM (#36212298)
Huh, if anything Google rarely returns walktroughts from steamcommunity.com for me and much prefers shoving hour long youtube videos.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by BizarroLand on Monday June 05, 2023 @11:55AM (#36198177)
The personal blocklist extension (and others, I'm sure) are good for removing sites from your results that you will never use.

Quora and Pinterest were the first to go for me.

[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by shadowgovt on Monday June 05, 2023 @09:29AM (#36195926)
One does wonder what the quality of organic results looks like in this era of curated content.
[ Parent | on HN ]
Re:Google No Longer Automatically Indexes Websites – WTF? by codyb on Monday June 05, 2023 @11:28AM (#36197649)
If you need an alternative DuckDuckGo seems _much_ cleaner to me... everytime I randomly inadvertently end up on Google I'm usually very turned off.
[ Parent | on HN ]