News for hackers. Stuff that matters.
Re:The Tragedy of Google Search by xnx on Friday September 22, 2023 @05:24PM (#37617983)
As the article mentions, the web has changed. It's very hard to return a list of good results when there are none. I've yet to see side-by-side comparisons where another search engine is doing a better job.

I've had "Search Generative Experience" (AI/LLM synthesized result/summary above the standard web results) turned on for awhile. It is definitely the future. To survive the flood of revenue optimized content (recipes being a familiar example), you need an active [software] agent working on your behalf to read content and make one straight thing out of the crooked timber of the web.

Re:The Tragedy of Google Search by A_D_E_P_T on Friday September 22, 2023 @05:26PM (#37618000)
The problem isn't just that the internet has become more SEO-oriented and encumbered with low-value clickbait websites -- it's also that Google doesn't listen to instructions the way it used to.

Google started slipping when it began assuming that it knows better than you do. For instance, when the exact search operator (" ") stopped working.

I'd take the Google of 2014 over the Google of 2023 without thinking twice, and I think that I'm not alone in this.

Re:The Tragedy of Google Search by endisneigh on Friday September 22, 2023 @05:27PM (#37618015)
Verbatim search still exists.

https://www.google.com/advanced_search

Re:The Tragedy of Google Search by A_D_E_P_T on Friday September 22, 2023 @05:32PM (#37618064)
Do you use it often? It doesn't work the way it used to. I can't count the number of times I've searched for a technical term -- in quotes -- and got results back for a subtly (sometimes not so subtly) different term.
Re:The Tragedy of Google Search by endisneigh on Friday September 22, 2023 @05:38PM (#37618135)
you have an example of this? using quotes is not the same as going to advanced search and saying use exact word or phrase. silly, yes I know.
Re:The Tragedy of Google Search by nostrademons on Friday September 22, 2023 @08:00PM (#37619310)
Quotes is different than verbatim mode. Compare:

[verbatim mode] https://www.google.com/search?q=unexpected+newline+in+compos...

[quotes] https://www.google.com/search?q=%22unexpected+newline+in+com...

The top two results are the same, but besides that verbatim mode seems way better to me.

Re:The Tragedy of Google Search by NegativeK on Saturday September 23, 2023 @10:48AM (#37623750)
I don't even bother with quotes anymore. When I'm searching for something technical from the normal interface, I'm almost always going to Tools -> All results dropdown -> Verbatim.
Re:The Tragedy of Google Search by eastern on Friday September 22, 2023 @10:01PM (#37619996)
I see that I can specifically ask for results that are in the Flash .swf format. Very convenient.
Re:The Tragedy of Google Search by intalentive on Friday September 22, 2023 @05:28PM (#37618029)
Agree. Also their results are often biased and I sometimes find myself going to Yandex.
Re:The Tragedy of Google Search by marginalia_nu on Friday September 22, 2023 @05:31PM (#37618062)
All search engines are inherently biased, since they perform a ranking task (which is a value judgement). They all encode (possibly uncessfully) the values of their creator. This is why having multiple search engines is important, as diversity in search options is the closes you'll get to objectivity.

(said the independent search engine creator with the same air of righteousness as when Plato argued philosophers should be kings)

Re:The Tragedy of Google Search by midhhhthrow on Friday September 22, 2023 @06:15PM (#37618467)
Yes but they’re also politically biased. Not to mention, the enormous amount of web content that is effectively banned from seArch engine results due to low prestige
Re:The Tragedy of Google Search by SV_BubbleTime on Friday September 22, 2023 @07:05PM (#37618939)
Congrats you are one step from re-inventing DogPile. Add AI and blockchain to pull in a ton of VC money.
Re:The Tragedy of Google Search by selimthegrim on Friday September 22, 2023 @05:31PM (#37618058)
I think Frank Zappa would chortle with irony that a Russian search engine returns his discographies uncensored in the here and now versus an American one.
Re:The Tragedy of Google Search by pascalxus on Friday September 22, 2023 @06:21PM (#37618509)
Absolutely. i so often do a search and it presumes to think it knows what i want rather than giving me what I ask for, sometimes even when I'm being quite specific. I will say it does a great job for programming stuff but a lousy job whenever I'm researching economics topics.
Re:The Tragedy of Google Search by kjkjadksj on Friday September 22, 2023 @07:21PM (#37619050)
Even programming stuff it does a lousy job. I use R a lot. I plot with the base graphics package because it works fine. Finding R information that does not involve the tidyverse library is so damn difficult on google search. Its bests to just go straight to the documentation since stackoverflow and other sites are pretty worthless with pretty much all answers requiring ggplot or dpylr even if they can be implemented with zero dependencies with the base tooling.

Same situation with python honestly. No I don’t want to use pandas, if I wanted better data wrangling I’d just wrangle in base R then import the wrangled data into python if need be instead of adding another dependency to my python code. Apparently this makes me a heretic based on google search results.

Re:The Tragedy of Google Search by 2OEH8eoCRo0 on Friday September 22, 2023 @06:46PM (#37618774)
More of the web is also behind paywalls or is on platforms like tiktok, instagram, twitter, etc. Less to scrape and index.
Re:The Tragedy of Google Search by rtsil on Friday September 22, 2023 @06:51PM (#37618816)
My most recent complaint is the disappearance of the pagination numbers.

Since I know the top results are mostly useless spams on some searches, I used to go to page 4 or 5 to find the actual results. Now to achieve the same result I have to click and scroll 5 times.

Re:The Tragedy of Google Search by charcircuit on Friday September 22, 2023 @07:21PM (#37619056)
>For instance, when the exact search operator (" ") stopped working.

It still works.

Re:The Tragedy of Google Search by Zambyte on Friday September 22, 2023 @07:46PM (#37619218)
It works, but it does something different than it used to. It used to only include results that actually contained the exact phrase you put within quotes. That is no longer the case.
Re:The Tragedy of Google Search by crazygringo on Friday September 22, 2023 @08:18PM (#37619436)
Now you just check "verbatim search" in the tools menu.

Works exactly how quotes used to.

Re:The Tragedy of Google Search by charcircuit on Friday September 22, 2023 @08:34PM (#37619535)
It works the same and it never only contained the exact phrase as Google ignores punctuation.
Re:The Tragedy of Google Search by unlikelytomato on Friday September 22, 2023 @08:57PM (#37619682)
This is not my experience. I search for exact phrases the same way I always have. Three to result does not contain the word I emphasized. This never happened before a few years ago. Now it happens daily.
Re:The Tragedy of Google Search by charcircuit on Friday September 22, 2023 @11:57PM (#37620476)
Are you sure the page doesn't contain it? It is not a simple as just visiting the page and searching for the thing you quoted as there may be punctuation between words or the text may be in the alt text of an image or somewhere else that is not obvious.
Re:The Tragedy of Google Search by unlikelytomato on Saturday September 23, 2023 @05:00AM (#37621687)
I am quite sure. The kinds of words in question would not even really make sense for "SEO gaming" to come into play. Typically, it presents with trying to match a word vs a phrase(I don't ever try to match phrases). I try matching smaller and smaller fragments of the word and still... nothing. I remember this working back when it was common for pages on gray websites to stuff giant blobs of text with juicy keywords out of view with janky html. Yet, somehow those pages were ranked very low and results were good. There is a clear degradation in quality.

Actually, it has happened with results that include hacker news discussions. I scrape through and expand EVERY comment on a 6 month old post and still find no partial or exact matches for my search keyword. It is very clear to me that Google has switched behavior. It used to flex on how awesome it was to bring you to an esoteric corner of the internet matching your exact query. Now, it tries to converge your queries into what it thinks you want(not what you asked for) and funnel you into large websites at the expense of utility.

Re:The Tragedy of Google Search by NegativeK on Saturday September 23, 2023 @10:50AM (#37623764)
It will substitute similar things, such as the word ubuntu for debian.

Which can be incredibly frustrating if you know the problem well enough to need a different answer.

Re:The Tragedy of Google Search by unlikelytomato on Sunday September 24, 2023 @12:19PM (#37634448)
This is a good point. I searched for "rename" today and the top result switched it to "name" instead. The TOP result completely changed the query. Infuriating.

I also love when my search is narrow enough that Google instead starts to return more generic pages for totally different programming languages instead of the smaller result set that matches exactly. It is very easy to observe as you explore a topic. At some point, you got a wall where any increase in search specificity seems to yield worse results than queries that helped you get to that point.

Re:The Tragedy of Google Search by NegativeK on Sunday September 24, 2023 @01:42PM (#37635340)
Yeah; that's a good point. Google's desire to provide popular/well known information sources is outweighing the ability for people to be specific. It's infuriating to be ignored as a user. Hence using verbatim constantly, which feels like a checkbox for "stop failing me".

Which... Why isn't Google learning that some users are constantly seeking out verbatim for results they'll click on and adjusting their search results to be more verbatim?

Re:The Tragedy of Google Search by xnx on Friday September 22, 2023 @05:41PM (#37618165)
I'll also had that a huge portion of genuinely good content is now inaccessible to Google in walled gardens like Discord, Facebook, Twitter, and TikTok.
Re:The Tragedy of Google Search by SV_BubbleTime on Friday September 22, 2023 @07:06PM (#37618952)
I remember google images before Getty and Pinterest ruined it.
Re:The Tragedy of Google Search by marginalia_nu on Friday September 22, 2023 @05:28PM (#37618030)
I don't think the problem, in general, is that no good results exist for any given query, more often the problem is that the good results are rarely well search engine optimized, and thus stand zero chance of making the search results.

Some people admittedly expect pretty weird websites to exist. Like who would even publish a good objective comparison between products in a specific category. But people make those queries looking for those types of results, and spammers generate such content to make bucks off the traffic. 20 years ago you'd have gotten nothing in most cases.

Re:The Tragedy of Google Search by mgaunard on Friday September 22, 2023 @06:48PM (#37618791)
There are a lot of people that genuinely make comparisons of specialty products.
Re:The Tragedy of Google Search by marginalia_nu on Friday September 22, 2023 @07:07PM (#37618955)
That sort of stuff is typically marketing.
Re:The Tragedy of Google Search by mgaunard on Saturday September 23, 2023 @08:24AM (#37622655)
Or simply just passionate people talking about a niche.

Consider coffee. You'll find hundreds of articles comparing the most high-end grinders. You even have PhDs writing white papers on the physics of coffee and how particle size distribution achieved by different grinders affects extraction quality. Those are genuine opinions from aficionados.

The same is true for all kinds of other things. The internet has a lot more than the auto-generated marketing crap you give it credit for.

Re:The Tragedy of Google Search by marginalia_nu on Saturday September 23, 2023 @10:25AM (#37623515)
This is true for things that are related to a hobby. There are many things where it basically doesn't exist. Nobody is writing articles comparing server colocation offerings near ${YOUR_CITY}, yet people search for them expecting them to exist.
Re:The Tragedy of Google Search by mgaunard on Saturday September 23, 2023 @04:37PM (#37627290)
That's not true, there are plenty of articles discussing these sorts of topics written by actual people.

Sniper in mahwah for example will discuss some of this for example from an HFT angle.

Re:The Tragedy of Google Search by marginalia_nu on Sunday September 24, 2023 @05:15AM (#37631391)
Such people exist in every city around the world?
Re:The Tragedy of Google Search by mgaunard on Monday September 25, 2023 @04:00PM (#37649743)
Finance markets are spread throughout the world and people spend a fair amount of money building efficient networks to route data between them faster than the competition, yes.
Re:The Tragedy of Google Search by marginalia_nu on Wednesday September 27, 2023 @09:56AM (#37674718)
Dude, there are fewer markets with a flash boys scene than there are cities in Kansas.
Re:The Tragedy of Google Search by bayindirh on Friday September 22, 2023 @05:27PM (#37618017)
So, how paid search engines like Kagi manages to bring useful results every time?

Its results are superior, with way less spam and SEO optimized content.

Re:The Tragedy of Google Search by mattnewton on Friday September 22, 2023 @05:39PM (#37618137)
I have no idea; but I speculate this is one area challengers have it easier because

a) there is no giant industry optimizing for ranking on Kagi. There are whole armies of people trying to game google search, meaning whatever heuristics they have would probably be rendered useless shortly if google were to adopt them.

And

b) Kagi has a much smaller, more technical userbase. So like the early web they can get away with a search experience tailored for those kinds of people and not millions of people powering on a smartphone as their first computer.

Re:The Tragedy of Google Search by bayindirh on Friday September 22, 2023 @05:47PM (#37618228)
The biggest power of Kagi is “you” are the algorithm. A page is optimizing for Kagi and providing bad results? Block them. You found a good website burrowed down? Raise/promote them.

Kagi works for you, with your input. It doesn’t do “advanced guessing” by implicitly and opaquely processing your interaction with it.

Re:The Tragedy of Google Search by mattnewton on Friday September 22, 2023 @06:04PM (#37618384)
Explicit control can work for ranking but can’t scale to the retrieval step of search - you aren’t going to sift through thousands of thousands of pages of potential results to give feedback on a meaningful portion of them, there must be some method of pulling pages to be ranked. This is where the gaming happens, with basically limitless spam domains that need to be filtered out. There just is no way Kagi is 100% relying on self reports for this.
Re:The Tragedy of Google Search by asdff on Friday September 22, 2023 @07:32PM (#37619140)
The whole idea of the model is to fit it on some reference so you can project it onto new data you haven't classified yet. So your first few weeks, you might get some spam sure. But eventually if you are optimizing the system by e.g. only ranking technical blogs with no seo bullshit, then technical blogs with no seo bullshit is what would fit best with that model among the new data and what you see on your results. That can't be gamed without knowing the sort of model fit you are generating and this fit is defined by the user.
Re:The Tragedy of Google Search by bayindirh on Saturday September 23, 2023 @02:52PM (#37626223)
The ranking is only relevant to you. It's part of Kagi's lenses and search tailored for you, by you approach.

I guess their "Universal Summarizer" is a part of their indexer, so they try to understand what you want and provide you more relevant results.

They probably have other signals and systems to detect spam and other unwanted things.

Re:The Tragedy of Google Search by amelius on Friday September 22, 2023 @06:33PM (#37618640)
Also Kagi is still in the phase of "let's give users what they want" instead of in the phase of "let's give shareholders what they want".
Re:The Tragedy of Google Search by bayindirh on Saturday September 23, 2023 @02:54PM (#37626242)
Since users are the ones who're financing Kagi, that phase will have no end until they become free and sell user data in a way.
Re:The Tragedy of Google Search by xnx on Friday September 22, 2023 @05:39PM (#37618146)
A lot of people on HN are very happy with Kagi, but I haven't seen examples of searches to understand where it is better. In a blind "taste test", I would be surprised if Kagi or Bing is preferred to Google for most searches.
Re:The Tragedy of Google Search by bayindirh on Friday September 22, 2023 @05:44PM (#37618201)
Well, all my daily searches, whether it be technical or daily mundane things return better results with less cruft than Google. Also, it notifies you if a page has excessive number of trackers.

Combine it with personal website ranking and lenses, it surpasses Google easily.

Also since it has no ads, there’s no noise.

I don’t use Microsoft products, so I can’t comment on Bing.

Re:The Tragedy of Google Search by fnordpiglet on Friday September 22, 2023 @05:46PM (#37618217)
I did side by side for a while and Kagi was consistently superior to Google and bing. I think it boils down to they don’t try to cross sell advertisements so they’re more oriented towards actually providing better results rather than better paying results.
Re:The Tragedy of Google Search by dharmab on Friday September 22, 2023 @05:57PM (#37618323)
Kagi's killer feature can't be blind tested, because Google and Bing don't have it. The ability to pin, raise or block sites in your results is a game changer.
Re:The Tragedy of Google Search by nostrademons on Friday September 22, 2023 @08:05PM (#37619344)
Interestingly, Google had that in 2008 (SearchWiki), but it got extremely low usage - enough that at a time when Google was serving 30K QPS, you could read the queries with SearchWiki interaction as they came in on a screen.

I suspect this is a niche feature for power-users only. HN is filled with power users, so they love Kagi, but it doesn't necessarily translate to the masses.

Re:The Tragedy of Google Search by dharmab on Friday September 22, 2023 @08:32PM (#37619524)
2008 didn't have Pinterest, fewer listicles, conspiracy websites were less mainstream, and mobile web was in its toddler phase.
Re:The Tragedy of Google Search by nostrademons on Friday September 22, 2023 @08:55PM (#37619669)
The feature was basically used exclusively to kill ExpertsExchange, so the problem existed, but yes, it wasn't quite as bad as today.
Re:The Tragedy of Google Search by janalsncm on Friday September 22, 2023 @08:53PM (#37619658)
Well, you would still compare Google as-is. It’s an apples to oranges comparison but you can still compare apples and oranges side by side.

I think it’s important to measure and quantify these things.

Re:The Tragedy of Google Search by dharmab on Saturday September 23, 2023 @12:18AM (#37620544)
"How do these products compare if I ignore their unique advantages" is not a useful measure.
Re:The Tragedy of Google Search by JohnFen on Friday September 22, 2023 @06:23PM (#37618532)
Do you need examples of searches? If people get more useful results from one search engine over another, that's all that matters.

I can't provide you with side-by-side examples because I don't compile lists of such examples. I'm searching to find stuff, not researching search quality. I just know that I have more problems getting good search results from Google over a couple of the other engines.

Re:The Tragedy of Google Search by xnx on Friday September 22, 2023 @06:31PM (#37618627)
Yes. I would love to see some actual examples. People have said that verbatim search doesn't work for them and I've never observed this, so I'm also skeptical of claims of dramatically better search results.
Re:The Tragedy of Google Search by JohnFen on Friday September 22, 2023 @06:43PM (#37618746)
So you think everyone is lying?
Re:The Tragedy of Google Search by ipaddr on Friday September 22, 2023 @07:41PM (#37619185)
No one can provide examples. But it doesn't mean they are lying. They may think it's better but actually isn't. Maybe it is worse but people are protective over it because their identity is connected to it's success.
Re:The Tragedy of Google Search by JohnFen on Monday September 25, 2023 @01:01PM (#37647056)
> They may think it's better but actually isn't.

If a user of service thinks a different service is better, then it is better for them by definition. "Better" is a subjective thing. If X seems better for me than Y, then it is.

Re:The Tragedy of Google Search by xnx on Friday September 22, 2023 @06:55PM (#37618853)
No. What they consider "better", I might consider "worse".
Re:The Tragedy of Google Search by dpkirchner on Friday September 22, 2023 @08:43PM (#37619600)
I can almost guarantee you that you'd find better results if you could block all results from spammy domains (without using an extension). It's practically a tautology.
Re:The Tragedy of Google Search by xnx on Friday September 22, 2023 @08:47PM (#37619628)
It's probably a good thing that just one week ago Google rolled out their "helpful content update" that incorporates a site-wide quality signal: https://developers.google.com/search/updates/helpful-content...
Re:The Tragedy of Google Search by dpkirchner on Friday September 22, 2023 @08:53PM (#37619657)
It looks like this uses some sort of automated system controlled by Google -- I am skeptical we'll see any improvements, given how bad results have been for so long (see all the SO and GH clones returned by Google for many examples). Who knows, though; I've certainly been wrong before.
Re:The Tragedy of Google Search by smcin on Friday September 22, 2023 @07:16PM (#37619018)
I cited [above](https://news.ycombinator.com/item?id=37618517) a Google search for

"Are US drivers licenses forgery-proof?" and "Can US Real-ID drivers licenses be forged?"

gave utterly irrelevant #1 or #2 hits: nothing to do with US or Real-ID (zero mentions), it's about New South Wales, Australia (lots of mentions of "NSW"). (although from US magazine Wired.com, but Google can still tell this article is totally irrelevant).

2) I previously in 2022 posted "Google search relevance fail: result for “Africa longitude” [https://news.ycombinator.com/item?id=30337563] where the #1 site hit and text result were the grossly misleading "Africa/Coordinates 8.7832° S, 34.5085° E", which is not a range, or even a centroid, just a junk location in southern Tanzania, where a (Russian?) SEO spam person seemed to have dropped a bogus pin, mislabeled "оф. 3, 27 ул. Хабаровская, Bila Tserkva, Russia". In Tanzania. 17 months later and same gloriously unfixed garbage results.

Other bizarre top hits were the small Ottawa e-commerce digital-map site https://www.mapsofworld.com/lat_long/africa.html, with the incomplete and inaccurate non-sentence "Africa's latitude and longitude lies between 9.1021° N, 18.2812° E." (That e-ecommerce store seems to have an autogenerated page with incorrect information on latitude and missing longitude for each country).

These are cases where Google itself 5 years ago would have done better. Or compare to Kagi.

Re:The Tragedy of Google Search by janalsncm on Friday September 22, 2023 @08:57PM (#37619684)
I’ve thought about making an account and spending the free quota on a side by side comparison. The problem is you need a representative sample of queries. The things I search might not be what you search or what my mom searches.
Re:The Tragedy of Google Search by nilespotter on Friday September 22, 2023 @06:25PM (#37618564)
80% good enough results and this [1] was all I needed to switch and not look back.

[1] https://kagi.com/privacy

Re:The Tragedy of Google Search by hutzlibu on Friday September 22, 2023 @05:47PM (#37618226)
"To survive the flood of revenue optimized content (recipes being a familiar example), you need an active [software] agent working on your behalf to read content and make one straight thing out of the crooked timber of the web."

So what do you think, will be the goal of all advertisers?

To get their recommendations baked into that AI. Direct bribing/revenue sharing with the manufactur of that AI. Or sneaking data into training sets via bribed employes and so on.

I don't trust black boxes and never will.

Re:The Tragedy of Google Search by bytefactory on Friday September 22, 2023 @07:08PM (#37618966)
Perhaps open-source AI models will be the our counterweight to commercial capture of AIs.
Re:The Tragedy of Google Search by asdff on Friday September 22, 2023 @07:27PM (#37619094)
Then these companies will just use closed source models.
Re:The Tragedy of Google Search by nine_k on Saturday September 23, 2023 @07:16AM (#37622194)
Right. That's why the agent should live under your control, and receive training from your search history.

Paid search is the future. Either you pay a subscription fee to something like Kagi, or you pay for hardware and electricity turn run your own "AI agent" locally.

Re:The Tragedy of Google Search by majormajor on Friday September 22, 2023 @06:01PM (#37618364)
I don't believe that anything Google does now - summarization included - will avoid being gamed just as effectively as their current search results have.

It's a negative-sum arms race, but if Google (or whoever) wasn't spending the money to fuel the arms race by passing on ad money, who would pay for Google to exist in the first place?

Re:The Tragedy of Google Search by JohnFen on Friday September 22, 2023 @06:33PM (#37618645)
I am convinced that a lot of the reason why Google search went downhill is that they started using more AI. Specifically, using it to try to interpret what I'm searching for rather than taking my word for it, which leads to irrelevant search results.

That's just my hypothesis, of course. I don't know for certain. But it does make me very skeptical about using AI to help search the web.

I want a robust set of search modifiers instead.

Re:The Tragedy of Google Search by beebeepka on Friday September 22, 2023 @06:34PM (#37618659)
I liked StartPage/Ixquick results much better than what Google used to show. Maybe they still do but wouldn't know as I stopped using it after they got acquired. Didn't/don't they rely on Google?
Re:The Tragedy of Google Search by c7b on Friday September 22, 2023 @06:35PM (#37618671)
> To survive the flood of revenue optimized content..., you need an active [software] agent working on your behalf

How long do you think it will be before that software agent's output will be just as riddled with commercialized content? I think we might have only a short window where we get to enjoy the fairly unadulterated output of the base LLMs by the large providers (yes, I'm aware that they're screening for safety already, but there could be a lot more coming). Imagine a dialogue like this: Q: What should I watch out for when buying new car tires?

A current answer might be something like: Here's a list of things to watch out for when buying car tires: A, B, C, D, E. Remember to always consult with specialists. Shall I explain more about the topic?

A future answer might look something like: According to experts like the professionals at [tire shop near you], here are the things you need to watch out for when buying car tires: A, B, C. You can use this coupon to get a 10% discount off your next purchase at [tire shop near you]. Shall I book an appointment for a free consultation at [tire shop near you] for you?

Re:The Tragedy of Google Search by autoexec on Friday September 22, 2023 @07:06PM (#37618943)
> you need an active [software] agent working on your behalf

I'd love to have one, but do any exist? SGE works on google's behalf, collecting data for google who will use what they learn from it against me. I'm guessing that until we can use common hardware to train/host an AI agent ourselves, we're stuck with terrible search results unless we're willing to hand over more and more data to others who want to exploit us, while pretending we can trust them to return results that are in our best interest instead of theirs. That's not a great future to look forward to.