News for hackers. Stuff that matters.
Re:Google No Longer Automatically Indexes Websites – WTF? by freitzkriesler2 on Monday June 05, 2023 @10:22AM (#36196557)
There was a Google search engineer on Reddit who claimed the opposite personally, Google is going down the trash but the alternatives aren't any better. Of course I can find it now, thanks Google.

I wish there was a search engine that ran like mid 2000s google but with a social media component so you can down vote SEO spammer blogs into oblivion.

Re:Google No Longer Automatically Indexes Websites – WTF? by lopis on Monday June 05, 2023 @10:29AM (#36196646)
Unfortunately, content farms can push new websites and blogs faster than you could ever downvote them. LLM are going to make that task increasingly easier. I've no idea how we're ever going to be able to search anything anymore using classic search engines. We either go back to website directories, or forward to AI generated content..
Re:Google No Longer Automatically Indexes Websites – WTF? by RajT88 on Monday June 05, 2023 @10:55AM (#36197043)
Web rings.

Think about it - some human element of trust and vouching for someone being added to the ring.

Re:Google No Longer Automatically Indexes Websites – WTF? by ddalex on Monday June 05, 2023 @11:14AM (#36197368)
Till you find out that most human will sell out ring links for a bit of cash with no problems.
Re:Google No Longer Automatically Indexes Websites – WTF? by RajT88 on Monday June 05, 2023 @11:42AM (#36197933)
Sure. It's a problem with peer-reviewed science journals even. There are no perfect solutions to monied interests bribing the curators.
Re:Google No Longer Automatically Indexes Websites – WTF? by mostlysimilar on Monday June 05, 2023 @11:38AM (#36197862)
Perhaps that is one additional layer of friction that will make human moderation / social voting feasible. The fire hose of AI trash content will come too rapidly for it to work at layer 1 (all content), but if the barrier to entry is a financial transaction to take over placement in a human-curated webring or directory it becomes easier to moderate / vote away the trash.
Re:Google No Longer Automatically Indexes Websites – WTF? by groby_b on Monday June 05, 2023 @12:04PM (#36198331)
Add a trust metric and chains of provenance. Bad ring link -> bad trust percolating up that chain. Little trust, your site isn't always shown as part of the ring. Too much loss of trust, you're out.

(Ultimately, this is a bad facsimile of human group behavior - all the way up to shunning people who deeply violate group norms. And I don't think it'll scale super-well. )

Re:Google No Longer Automatically Indexes Websites – WTF? by gregoryl on Monday June 05, 2023 @12:52PM (#36199180)
That's pagerank, right? The trust was built from href votes.
Re:Google No Longer Automatically Indexes Websites – WTF? by groby_b on Monday June 05, 2023 @02:40PM (#36200999)
Except there's no provenance or root of trust. There is (IIUC) no back-propagation of a penalty if sites violate trust, just an overall observational measure.

And I'd still say pagerank did work really well in an Internet where there was overwhelmingly trust. But in a world where default-trust is a bad stance, I believe there needs to be an equivalent of what "You can trust X" does in small in-person groups. (Or, alternatively "Sheesh, X went off and just destroyed all trust")

I do think it'll need to be more than a single metric, too. Trust is multidimensional by topic(E.g. "I trust the NYTs data science folks, I have zero trust for the OpEds"), and it is somewhat personal. (E.g. I might have experienced X lying to me, while they've been 100% honest to you - maybe in/outgroup, maybe political alignment, maybe differing beliefs, etc. Ultimately, what we call trust in an indirect situation is "most of my directly trusted folk vouch for that person)

Re:Google No Longer Automatically Indexes Websites – WTF? by nunuvit on Monday June 05, 2023 @11:43AM (#36197948)
Keyservers. You decide which keyservers to register with and to trust for verifying others. Browsers would handle en-decryption automatically and allow you to flag, filter, or finger (in the Unix sense).
Re:Google No Longer Automatically Indexes Websites – WTF? by CamperBob2 on Monday June 05, 2023 @02:02PM (#36200345)
No, they can't. Or at least, they don't. I see the same trash-fire sites on Google all the time. Google just DGAF.
Re:Google No Longer Automatically Indexes Websites – WTF? by dcow on Monday June 05, 2023 @10:29AM (#36196648)
Kagi is an alternative and it is worlds better. Try it out!

https://kagi.com

Re:Google No Longer Automatically Indexes Websites – WTF? by freitzkriesler2 on Monday June 05, 2023 @10:36AM (#36196741)
Checking this out, thanks mate.
Re:Google No Longer Automatically Indexes Websites – WTF? by rolisz on Monday June 05, 2023 @11:07AM (#36197242)
I'm also a huge fan of Kagi. I've been a paying user since they launched the paying subscription. Really happy with it!
Re:Google No Longer Automatically Indexes Websites – WTF? by dhc02 on Monday June 05, 2023 @11:11AM (#36197320)
Love kagi. The first time I got the "your payment was successful" notification I felt like I'd never get that much value out of it. But now, a few months later, I feel like I could never go back.
Re:Google No Longer Automatically Indexes Websites – WTF? by mordae on Monday June 05, 2023 @11:35AM (#36197802)
Though a subscriber myself, Kagi doesn't really add results, does it? It merely weeds out the trash for you. So you can get to the bottom of search results.
Re:Google No Longer Automatically Indexes Websites – WTF? by anonred on Monday June 05, 2023 @11:54AM (#36198164)
Just being able to block spammy Stack Overflow clones from ever appearing in the results is worth the price of admission for me.
Re:Google No Longer Automatically Indexes Websites – WTF? by saltysalt on Monday June 05, 2023 @12:13PM (#36198464)
Here is another, just launched: https://greppr.org/
Re:Google No Longer Automatically Indexes Websites – WTF? by Minor49er on Monday June 05, 2023 @03:34PM (#36202290)
Looks promising, though I noticed that it doesn't encode queries properly when searching. For example, if you go to the homepage and search for "../robots.txt", you'll be redirected to the site's own robots.txt file
Re:Google No Longer Automatically Indexes Websites – WTF? by saltysalt on Monday June 05, 2023 @04:15PM (#36202910)
Thank you kindly for testing, I'll need to fix that one.
Re:Google No Longer Automatically Indexes Websites – WTF? by anderber on Monday June 05, 2023 @07:42PM (#36205966)
Don't they use Google for their results? I'd imagine they'd run into the same issues the article is pointing out.
Re:Google No Longer Automatically Indexes Websites – WTF? by dcow on Tuesday June 06, 2023 @10:45AM (#36213614)
No? At least not like you are implying. Kagi queries multiple data sources and synthesizes results. This means Google’s failure to index does not impact Kagi in the same way as it would DDG (with Bing).
Re:Google No Longer Automatically Indexes Websites – WTF? by winternett on Monday June 05, 2023 @10:26AM (#36196606)
I miss AltaVista so much... It was no frills and only based on page content.
Re:Google No Longer Automatically Indexes Websites – WTF? by BryantD on Monday June 05, 2023 @10:28AM (#36196630)
Those of us who worked there thank you!
Re:Google No Longer Automatically Indexes Websites – WTF? by mech422 on Monday June 05, 2023 @10:34AM (#36196713)
I loved me the 'near' keyword - thanks!
Re:Google No Longer Automatically Indexes Websites – WTF? by masukomi on Monday June 05, 2023 @11:32AM (#36197746)
I both appreciated Alta Vista, and appreciated its office space in Littleton ( i think ) when i worked in it after its passing. ;)
Re:Google No Longer Automatically Indexes Websites – WTF? by lispy4 on Monday June 05, 2023 @12:08PM (#36198410)
Loved everything about Alta Vista, including the logo, and the UI.

I miss 90s Internet in general. It wasn't the ugly battleground and desolation planet that the current net has become.

Re:Google No Longer Automatically Indexes Websites – WTF? by fragmede on Monday June 05, 2023 @12:13PM (#36198469)
Remember Guestbooks? you'd visit a website, and volunteer your name and which country you were from and leave comments. And it wouldn't be a cesspool of spam and porn and XSS attacks? How quaint!
Re:Google No Longer Automatically Indexes Websites – WTF? by lispy4 on Monday June 05, 2023 @12:35PM (#36198861)
Oh gosh, yes! And reading the guestbook was always so fun. An elderly friend of mine passed away in 2018, and in doing a (google) search of him, I found guestbooks he'd signed 20 years ago.
Re:Google No Longer Automatically Indexes Websites – WTF? by zorked on Monday June 05, 2023 @02:29PM (#36200809)
You would look for a thing and the first five pages were random mailing list discussion archives discussing how the thing was 5 years before... Altavista was impressive, but there is a reason why it went away.
Re:Google No Longer Automatically Indexes Websites – WTF? by throwaway894345 on Monday June 05, 2023 @11:03AM (#36197177)
I used DDG for a while, but DDG's quality fell precipitously a few years ago (similar issues where it ignores quotes and won't find pages even if you search for the title string exactly, etc) and I eventually came back to Google which has also been increasingly frustrating.

> I wish there was a search engine that ran like mid 2000s google but with a social media component so you can down vote SEO spammer blogs into oblivion.

There's no way this won't get abused, but the SEO stuff is out of control. Not even spammer blogs, but if you have a quick question like "how do I check tire pressure" you will only get articles that start with a treatise on the entire history of car tires and the answer is deeply buried somewhere in the article. My guess is that Google sees that we're on the page for a longer time than we would spend on pages that just return the answer, and they assume that "more time on page" == "better content" or something.

Re:Google No Longer Automatically Indexes Websites – WTF? by red_admiral on Monday June 05, 2023 @11:13AM (#36197349)
The tire pressure query is exactly the kind of thing that AI should be able to handle easily, though. At which point google has an incentive to sort their competitiveness out.
Re:Google No Longer Automatically Indexes Websites – WTF? by AshamedCaptain on Monday June 05, 2023 @11:52AM (#36198124)
DDG has become ridiculous. They seem to be merging "local", geoIP based results no matter what country I select on the region list (or I disable it). Very often completely unrelated stuff (but local) appears on the 5th or 6th result, midway the first page.

Most egregiously I will search for something very rare (e.g. about programming) and DDG will return me results regarding my city's tourist/visitor info. It's as if it just keeps ignoring words from the search prompt that return no results until it runs out of keywords then it's just the geoIP results.

Re:Google No Longer Automatically Indexes Websites – WTF? by account42 on Tuesday June 06, 2023 @07:09AM (#36210925)
I hate this forced localization so much and its everywhere. The internet used to be a place where you would actually encounter stuff outside your locale.
Re:Google No Longer Automatically Indexes Websites – WTF? by pixodaros on Monday June 05, 2023 @12:17PM (#36198542)
That is because DuckDuckGo started relying almost entirely on Bing for their regular search results after first Yahoo gave up maintaining its own index then Yandex became part of a natio non grata leaving them to choose between partnering with Bing and partnering with Google or creating their own index https://help.duckduckgo.com/duckduckgo-help-pages/results/so...
Re:Google No Longer Automatically Indexes Websites – WTF? by mostlysimilar on Monday June 05, 2023 @11:42AM (#36197937)
> I wish there was a search engine that ran like mid 2000s google but with a social media component so you can down vote SEO spammer blogs into oblivion.

I want this too, but I think an often understated aspect of this issue is that by this point Google has absolutely trashed the web of that era. In these threads people will say "the content you want isn't out there, it's all on social media now" -- and they're largely right, but I think Google is the party most responsible for mutilating the web to the state it is in now, and users fled to social media partly because it seemed like a safe haven.

What we need is a concentrated effort to rebuild the web. Take the best parts of what we've learned and combine with the best parts of what we've left behind and try to build something better, for humans, not for advertisers and hyper-capitalists.

That will take time, energy, and people who remember what we lost and believe we can build something better. A better search engine alone is not enough.

Re:Google No Longer Automatically Indexes Websites – WTF? by JustLurking2022 on Monday June 05, 2023 @12:32PM (#36198798)
The web changed when every kinda slimy business bro realized they could monetize gaming search results. No matter what your fantasy web looks like, be assured, people will game it to the point it's not what you intended.
Re:Google No Longer Automatically Indexes Websites – WTF? by mostlysimilar on Monday June 05, 2023 @12:37PM (#36198902)
If I take that viewpoint on everything I might as well live as a recluse in the woods and avoid people altogether. I have to believe that there are enough of us are out there that genuinely want to build better things for people.
Re:Google No Longer Automatically Indexes Websites – WTF? by noizejoy on Monday June 05, 2023 @05:43PM (#36204267)
The web, just like the real world isn’t static. Becoming and staying intellectually, emotionally and physically mobile may be the only long term strategy to avoid ending up in one or the other dystopia, sooner or later.

When rates of change were slower, you might only have to “move” once in your life, but with increasing rates of change in our human experience, staying nimble is arguably of ever increasing importance.

Re:Google No Longer Automatically Indexes Websites – WTF? by joeythedolphin on Wednesday June 07, 2023 @01:05AM (#36223062)
Yes. This is like water -- keep it moving, find fresh streams.
Re:Google No Longer Automatically Indexes Websites – WTF? by JustLurking2022 on Monday June 05, 2023 @06:41PM (#36205150)
And my point is that there are probably a lot of those motivated people working on the problem today. You make it out as though we've arrived at this state by either lack of effort or competence by Google/Microsoft. My guess is that every time they change the algorithm, the spammers adapt too. That's inevitable and would be just as much of a challenge for your supposed utopia. If you have some secret they don't, there's certainly plenty of money to be made.
Re:Google No Longer Automatically Indexes Websites – WTF? by yomlica8 on Monday June 05, 2023 @12:46PM (#36199076)
It seems like google hardly returns results from traditional forums or blogs which has probably accelerated their decline artificially.
Re:Google No Longer Automatically Indexes Websites – WTF? by Falkon1313 on Monday June 05, 2023 @08:44PM (#36206582)
Largely right, but actually a lot of that stuff is still out there. The personal and hobby pages, forums, blogs, etc.

Google just doesn't know that they exist anymore, or rather doesn't want us to know, because those sites are not commercial enough or big enough.

Almost without fail, no matter what you search for, it tries its best to turn it into a search for a product or service. And those content oriented websites don't fit that, so it just pretends they don't exist.

Re:Google No Longer Automatically Indexes Websites – WTF? by fallingknife on Monday June 05, 2023 @11:48AM (#36198045)
What I want is a "serious mode" that makes it favor primary sources, peer reviewed papers, and raw data. When I search for economic data, I don't want a million news articles referencing pieces of it. I want the raw data release. When I search for some video going viral, I don't want a million videos of journalists talking and showing clips. I want the full raw video.
Re:Google No Longer Automatically Indexes Websites – WTF? by melagonster on Tuesday June 06, 2023 @05:55AM (#36210431)
Google scholar?
Re:Google No Longer Automatically Indexes Websites – WTF? by joeythedolphin on Wednesday June 07, 2023 @12:59AM (#36223022)
Beautifully said! As a thinker of philosophy, I have come to understand that our clip-society is based by design. People can express power over others if they tell you a construction and then show a clip to support it. They really don't want you to see the source/what it is/the truth. They want you want you see what they show you. This problem is accelerating in western societies and it is a fundamental problem of human nature. Journalism is the healthy expression and what we see in today's media is the sickly end.
Re:Google No Longer Automatically Indexes Websites – WTF? by marginalia_nu on Monday June 05, 2023 @01:15PM (#36199543)
> I wish there was a search engine that ran like mid 2000s google but with a social media component so you can down vote SEO spammer blogs into oblivion.

This is sort of what I've been trying to do with Marginalia Search, except I don't really believe a voting system would work. It's far too easy to manipulate. Been playing with the thought of having something like an adblock-list style system where domain shitlists can be collaborated on and shared without being authoritative for the entire search engine.

My search engine is still pretty rough around the edges and limited but I think it works well enough to demonstrate the idea has some merit.

Re:Google No Longer Automatically Indexes Websites – WTF? by Tommstein on Monday June 05, 2023 @06:07PM (#36204665)
> Been playing with the thought of having something like an adblock-list style system where domain shitlists can be collaborated on and shared without being authoritative for the entire search engine.

Even just personal shitlists would be golden and make just about everyone happy.

Re:Google No Longer Automatically Indexes Websites – WTF? by freitzkriesler2 on Monday June 05, 2023 @08:54PM (#36206657)
Will check it out, thanks bud.
Re:Google No Longer Automatically Indexes Websites – WTF? by hnick on Monday June 05, 2023 @09:49PM (#36207052)
Something I've wanted (which probably exists as an extension in Chrome?) for Google searches is a simple blacklist. Just a little button and confirmation next to a result, telling it to never show this blog-spam-ad-laden-SEO-mess of a page to me ever again. Maybe it's an uphill battle, but for some niche topics (like certain games) there are some sites I keep having to scroll past and sometimes accidentally click that are written in SEO-speak and say a lot without saying anything at all.
Re:Google No Longer Automatically Indexes Websites – WTF? by eadler on Tuesday June 06, 2023 @11:54PM (#36222636)