News for hackers. Stuff that matters.
Re:Google No Longer Automatically Indexes Websites – WTF? by danwee on Monday June 05, 2023 @10:05AM (#36196332)
Google Search is in decline for the users who know how the internet works. For my mother, the internet is still Google (and would remain like that no matter what). For me, for some friends of mine, and for many of my colleagues who know more or less how things work on the web, Google Search is just in free fall: we use it as last resort, but as other search engines (or other tools, like ChatGPT) improve over time, Google Search would just disappear from our bookmarks.

I know that many of you would say that it doesn't really matter what "hackers" think about Google Search, that all that matters is that the majority of the non-tech-savvy users still think Internet == Google. Well, let's talk again in 5 years.

Re:Google No Longer Automatically Indexes Websites – WTF? by mjburgess on Monday June 05, 2023 @10:35AM (#36196725)
I certainly start with google far less -- I now, very often, stray little from my bookmarks bar. The first of which is "New Chat" (ChatGPT)

I needed some somewhat obscure API information recently, and ChatGPT had it -- a testament to how much ChatGPT really is just a compression of "everything ever digitised"

Re:Google No Longer Automatically Indexes Websites – WTF? by devjab on Monday June 05, 2023 @11:15AM (#36197390)
Does ChatGPT “have it”? I thought it made stuff up and that the magic it does is being very good at making the stuff up.

We’ve had it come up with solutions using entirely made up functions. They had names that sounded like something Microsoft would’ve put into .Net, but they were entirely made up. As in, they had never existed in any version of .Net ever.

So as much as I like it, I’m treating it with more caution than google results. Honestly though, most of the time it’s frankly faster to just read the damn manual and figure out things for yourself. I don’t say that as some sort of anti-prompt-programming purist, but wading through GPT responses is as about as hopeless as wading through the gazillion medium, dev.to, stack overflow and whatever else people post their useless stuff on. 10 years ago if I needed to do some obscure Sharepoint programming (waaaaaay out of my field) I could realistically make something work with the help of Google, today the same thing is frankly completely impossible.

Re:Google No Longer Automatically Indexes Websites – WTF? by esperent on Monday June 05, 2023 @11:29AM (#36197679)
> Does ChatGPT “have it”? I thought it made stuff up and that the magic it does is being very good at making the stuff up

ChatGPT is just a frontend UI and refers to two different models (and different tunings of these models over time I guess).

GPT3.5 is kind of useful but it makes up stuff just enough that you can't really trust it and spend so much time verifying it's hard to say whether you saved time vs the old way. But it still produces mostly not made up stuff.

GPT-4, still not perfect, is a game changer though. It's what people generally mean when they talk about ChatGPT. There's far far less hallucination going on (not zero though). There's several ways to access it: phind, bing, ChatGPT. But I still give ChatGPT is the best.

Re:Google No Longer Automatically Indexes Websites – WTF? by lacy_tinpot on Monday June 05, 2023 @11:35AM (#36197801)
>I thought it made stuff up and that the magic it does is being very good at making the stuff up.

I don't get this problem. Google search doesn't give you answers, it's not reliable, and you still have to double check the answers your.

Remember that? Any information found on the internet used to be assumed to phony too.

Re:Google No Longer Automatically Indexes Websites – WTF? by bodge5000 on Monday June 05, 2023 @11:48AM (#36198036)
Its actually quite interesting just how good it is at making things up. When I first started using it I didn't realise it even could, so when I got a non-existent GDScript function, I started to prod it to see where it came from. It was able to explain the function, tell me when it was added (of course Godot is open source so it would in theory have access to this) and even the commit hash used to add it, and all of it sounded very plausible. It was only when I pointed out that it doesn't exist that it admitted it.

Admittedly that was on GPT3, I haven't tried GPT4 as I can't afford it at the moment. No doubt its better, but I'm not sure how much so.

Re:Google No Longer Automatically Indexes Websites – WTF? by mjburgess on Monday June 05, 2023 @11:52AM (#36198111)
It's always making things up. It's fundamentally a coincidence when it gets it right.

That's the nature of associative statistics -- and why this talk of "hallucination" is more marketing PR.

We hallucinate in that our reconstructions of our environment can be caused by internal states (eg., dreaming) -- whereas veridical perceptual states are usually caused directly by the environment.

Here, it's states (ie., the statistical averaging process over its training data) is NEVER caused veridically -- ie., its prompts are never caused by the environment.

Re:Google No Longer Automatically Indexes Websites – WTF? by mjburgess on Monday June 05, 2023 @11:50AM (#36198069)
The more ChatGPT is just a remembering of its content, the better it is. In this case it had clearly remembered just that API (obscure educational API) -- with weird parameter key dictionaries etc. that arent in anyway some Intelligent Generalisation (oh wow! everyone fund this!!11!)

Insofar as it isn't just a regurgitation of "the better ebooks, blogs and docs", the worse it is.

This is why, when prompting ChatGPT, i'm more often aiming to have it use examples (etc.) that have a high likelyhood of exact data in its training set.

Consider eg., the prompt, "write tic-tac-toe in javascript using html and canvas" to "write duck hunt in javascript using html and canvas"

The latter is extremely hard to get out of it, with many prompts -- the former, immediate and perfect.

Why? because there's many examples of tic-tac-toe.

Re:Google No Longer Automatically Indexes Websites – WTF? by falcrist on Monday June 05, 2023 @01:03PM (#36199372)
GPT doesn't give correct answers. It gives answers that sound correct.

Those correct-sounding answers are often actually correct, but this is more coincidence than design. Anything it says is suspect, and should be fact-checked.

Re:Google No Longer Automatically Indexes Websites – WTF? by okdood64 on Monday June 05, 2023 @12:04PM (#36198343)
How did you know it was correct? That that API wasn't deprecated? Or there was a better one for your use case?
Re:Google No Longer Automatically Indexes Websites – WTF? by mjburgess on Monday June 05, 2023 @12:32PM (#36198800)
I also had api examples open
Re:Google No Longer Automatically Indexes Websites – WTF? by okdood64 on Monday June 05, 2023 @10:34AM (#36196707)
I'm pretty sure you could find doom & gloom talk about Google on HN 5 years ago.
Re:Google No Longer Automatically Indexes Websites – WTF? by pil0u on Monday June 05, 2023 @10:58AM (#36197083)
Only 7 years ago, the world was astonished with the release of AlphaGo. This was an algorithm dedicated to one very specific task of winning a game.

The most recent game being played is language, a game that Google has played alone, until now.

Doom & gloom talk about Google 5 years ago was probably unjustified and rightfully dismissed. Today, I'm not so sure about that.

Re:Google No Longer Automatically Indexes Websites – WTF? by zarzavat on Monday June 05, 2023 @11:02AM (#36197161)
That’s because it’s been progressively getting worse for years. For example, when they launched Google+ they stopped supporting the + operator in favour of “quotes” and people complained about that. Of course now the “quotes” don’t work either.

Google used to be amazing. If you remembered a set of words that were on a page, you could enter those words and it would find matching pages. It’s been broken for years and keeps getting more broken.

Re:Google No Longer Automatically Indexes Websites – WTF? by okdood64 on Monday June 05, 2023 @12:02PM (#36198304)
Broken?

https://www.google.com/search?q=%22may+have+been+less+attrac...

I just took a random snippet from Gergely's blog and using quotes I found it.

Look there's many issues with Google Search in 2023, but these nitpicks that aren't neccesarily true aren't productive to the conversation.

Re:Google No Longer Automatically Indexes Websites – WTF? by lazypenguin on Monday June 05, 2023 @09:43PM (#36207010)
Yes it's broken, try searching 'python abs "Return the absolute value of a number"'. That string is directly lifted from the Python docs for the `abs` function. The official Python documentation does not appear for me until the 3rd page and the "featured snippet" is from some random website called flexiple.com
Re:Google No Longer Automatically Indexes Websites – WTF? by rafark on Monday June 05, 2023 @09:51PM (#36207064)
It’s not broken. I use the exact match operator “” all the time and it works pretty much every time. Selecting verbatim works even better.

https://i.ibb.co/D8VfKxc/75-FC1-FC6-DFD7-4-E2-A-A717-DBD8463...

Re:Google No Longer Automatically Indexes Websites – WTF? by lazypenguin on Monday June 05, 2023 @10:55PM (#36207502)
I didn’t search “python docs” I searched “python abs”
Re:Google No Longer Automatically Indexes Websites – WTF? by rafark on Tuesday June 06, 2023 @12:12AM (#36207999)
Well that’s the problem then. You could say your problem has more to do with the relevancy of the results than the exact match operator (I personally don’t mind in this case since for the few times I’ve had to write python I’ve found the official docs not as useful as I’d expect).

As I said, I use that operator all the time and it works. I’d be the first to complain when it doesn’t. For instance, the - operator stopped working in YouTube a while ago, unfortunately.

Re:Google No Longer Automatically Indexes Websites – WTF? by micromacrofoot on Monday June 05, 2023 @11:08AM (#36197255)
it's been declining for longer than 5 years, especially since they've removed discussion result filtering
Re:Google No Longer Automatically Indexes Websites – WTF? by jsight on Monday June 05, 2023 @11:40AM (#36197892)
It reminds me of when people would talk doom for IE based upon some tiny uptick in Firefox users and the relatively stagnation of IE as a browser.

They were obviously wrong.

Then Chrome happened. If Google doesn't change some fundamental things, they'll be the next one to fall for similar reasons.

I'd be less concerned for them if I didn't see signs of decline with Android as well. It is a key component for them to control.

Re:Google No Longer Automatically Indexes Websites – WTF? by matthewbauer on Monday June 05, 2023 @12:37PM (#36198910)
That reminds me of that quote about bankruptcy from The Sun Also Rises:

> “How did you go bankrupt?” Bill asked.

> “Two ways,” Mike said. “Gradually and then suddenly.”

It felt like that’s how IE lost its dominance.

Re:Google No Longer Automatically Indexes Websites – WTF? by genezeta on Monday June 05, 2023 @02:05PM (#36200392)
In 2004 IE had 94% share of the browser market.

There were "tiny upticks" but, going slowly and steadily, in August 2008 Firefox had gained about 30-33% and IE was down to about 60%. And then (September 2008) Chrome happened.

Re:Google No Longer Automatically Indexes Websites – WTF? by rightbyte on Tuesday June 06, 2023 @01:36AM (#36208561)
In hindsight Chrome was worse for the internet than IE ever was.

A spyware and ads company fooling devs and users alike.

Re:Google No Longer Automatically Indexes Websites – WTF? by account42 on Tuesday June 06, 2023 @09:04AM (#36212029)
Well a lot of those devs seem to like the spying as long as they themselves have access to (some of) the data.
Re:Google No Longer Automatically Indexes Websites – WTF? by yomlica8 on Monday June 05, 2023 @02:13PM (#36200529)
Firefox did most of the early damage to IE all it's own. Taking IE down from an insane 90+% marketshare position and steadily eating up 30% marketshare by the time Chrome even really showed up. And it didn't even have to stealth install itself with every adobe flash update to do it.

https://gs.statcounter.com/browser-market-share/all/worldwid...

Re:Google No Longer Automatically Indexes Websites – WTF? by jeffbee on Monday June 05, 2023 @11:21AM (#36197518)
I think it's actually a "u"-shaped curve. For people with no idea, Google works great. For people with vague and mostly inaccurate knowledge of how to index and search the web, it appears to suck. For people who actually know, it once again seems good.
Re:Google No Longer Automatically Indexes Websites – WTF? by nix0n on Monday June 05, 2023 @03:30PM (#36202225)
> For people who actually know, it once again seems good.

If the way to make it work is to use "site:", to me that doesn't count as "searching the web" anymore.

Re:Google No Longer Automatically Indexes Websites – WTF? by evouga on Monday June 05, 2023 @11:35AM (#36197797)
There was a magic period of time lasting about a decade when internet search Just Worked. If it was on the public web Google would find it. Search worked so well I took it for granted.

Today I avoid Google search at all costs, and use it mainly to find Wikipedia pages or to search Reddit.

The Googlers blame SEO and there is some truth to that; but Google has on retainer a huge stable of the world’s best-paid engineers, and still couldn’t be bothered to invest in their flagship service.

Re:Google No Longer Automatically Indexes Websites – WTF? by nvarsj on Monday June 05, 2023 @11:45AM (#36197978)
The flagship service is ads not search.
Re:Google No Longer Automatically Indexes Websites – WTF? by MarcellusDrum on Monday June 05, 2023 @11:54AM (#36198152)
Well good luck showing people ads when people start using ChatGPT for all their internet searches.
Re:Google No Longer Automatically Indexes Websites – WTF? by ptmcc on Monday June 05, 2023 @12:03PM (#36198321)
It'll be so easy to inject ads into things like ChatGPT, and it'll even look "natural" to most readers.
Re:Google No Longer Automatically Indexes Websites – WTF? by mattlondon on Monday June 05, 2023 @01:33PM (#36199846)
It would be trivial to insert banner ads, or even "native ads" in the response.

LLMs are probably even better suited for monetisation since they have a better understanding of what people want, so a better ad can be shown that is more likely to be clicked.

Do you think people were losing their shit when ChatGPT went into bing simply because it might be taking market share from Google? No - people were losing their shit because it meant the ads in search are going to be turbo-charged and so that is why the share prices are surging (GOOG up 40% since Jan): more ad revenue from "better" ads shown to users.

Re:Google No Longer Automatically Indexes Websites – WTF? by survirtual on Monday June 05, 2023 @11:45PM (#36207807)
People still need products to solve problems. When you ask an AI "I have X problem, what products could I use to resolve it?" that is a natural and ethical time to include advertisers into the equation. The LLM can be trained on product specifications, details, and relevant uses, as well as plug in to a review database. Companies can pay to be included in the results of possible solutions and the AI can use the available information to make specific recommendations based on real data.

Hopefully, the future will completely prune intrusive, non-consensual advertising completely and any companies that inject thoughts into our minds will fail.

Re:Google No Longer Automatically Indexes Websites – WTF? by account42 on Tuesday June 06, 2023 @08:54AM (#36211896)
> People still need products to solve problems.

For some problems. For many more you don't need a "product" at all but advertising still wants to sell you one.

> that is a natural and ethical time to include advertisers into the equation

Not at all. Ads means showing the product of whoever is paying the most or at least preferring paying products over others. Ad-free suggestions means showing the product best suited for the task. If those two match you are defrauding the advertiser by making them pay to show what you would have shown without ads. If they don't match you are degrading the service for the users.

Re:Google No Longer Automatically Indexes Websites – WTF? by survirtual on Tuesday June 06, 2023 @11:42AM (#36214564)
Imagine a world where we prioritize optimizing for the best human experience instead of doing the bidding of people with capital.

You can do it, I believe in you.

Re:Google No Longer Automatically Indexes Websites – WTF? by MichaelZuo on Monday June 05, 2023 @11:52AM (#36198122)
I agree, Google up until ~2013 felt magical and nowadays even news.ycombinator.com, a site frequented by many search engineers, isn't fully indexed anymore, not even 80% indexed for some obscure search terms.
Re:Google No Longer Automatically Indexes Websites – WTF? by Pooge on Monday June 05, 2023 @12:33PM (#36198821)
Which search engine do you use?
Re:Google No Longer Automatically Indexes Websites – WTF? by lampiaio on Monday June 05, 2023 @12:42PM (#36199013)
I just checked my search history, and about 80% of my Google queries contain "site:reddit.com". It's becoming harder and harder to find real content.
Re:Google No Longer Automatically Indexes Websites – WTF? by xerxesaa on Monday June 05, 2023 @12:33PM (#36198835)
Genuine question: if Google is a last resort, what are your more preferred alternatives? I've tried many search engines and haven't really succeeded in finding a better alternative. Wondering if I'm missing something.
Re:Google No Longer Automatically Indexes Websites – WTF? by codalan on Monday June 05, 2023 @12:48PM (#36199095)
Kagi (paid) and Brave search have both been good to me.

I never really liked DDG or Bing. They both suffer from the same SEO gaming that Google does.

There are also a number of smaller search engines that have been posted to HN that are kind of interesting for certain niches.

I think what will happen over the next five years is that instead of Google being the one-stop shop for search, it will be a number of smaller players + the different chatbot engines like ChatGPT and Bard and others.

Re:Google No Longer Automatically Indexes Websites – WTF? by rocho on Tuesday June 06, 2023 @01:02AM (#36208312)
I tried many alternatives, and finally settled on Kagi Search. It's a paid option but it's well worth it. They also have tools like FastGPT which I'm using more and more. It's a GPT model that has access to their search results. You can't use it for conversation like you'd do with ChatGPT, but for searching and summarizing it's amazing.
Re:Google No Longer Automatically Indexes Websites – WTF? by romusha on Monday June 05, 2023 @10:22PM (#36207267)
What search engines would you recommend? I'm having difficulty using google, its just practically useless these days. ChatGPT is much much better but I'd like to know other tools
Re:Google No Longer Automatically Indexes Websites – WTF? by jesterson on Tuesday June 06, 2023 @02:38AM (#36208990)
> Well, let's talk again in 5 years.

Every time new fad appears, I say this. And I say the same about chatgpt. Never failed so far.

So yea, let's talk again in 5 years.

Re:Google No Longer Automatically Indexes Websites – WTF? by gorbachev on Tuesday June 06, 2023 @07:19AM (#36211007)
Google == Internet will continue as long as Google remains the default search engine on the most popular browsers.