> I really wish a search engine would empower me to provide a URL ban list that gets applied server-side instead of filtering on the front end (if anything).

The irony is that Google wants all our information "to improve our user experience", except they don't want our ban lists.

Goggles is coming. https://brave.com/wp-content/uploads/2021/03/goggles.pdf

This is an exciting idea, but the style of the paper is off-putting. It's written in the style of an academic paper, while clearly eschewing associated norms like not giving blatant opinion or waxing philosophical.

"The rationale is not to customize the ranking according to the implicit interests of the user, but to offer a mechanism to define multiple rankings, plural, open and explicit, for only if it is so, can it be trusted."

Please put opinions in a blogpost and uphold the (reasonable) norms of the research community, for only if it is so, can your work be trusted.

pinterest. Bane of all image searches.

I use the uBlackList addon in Firefox to block Pinterest results from Google search results with the following regular expression:

    /pinterest\..*/

It also appends "block this site" links to all search results.

Also Tenor. They've taken over just about all animated GIF results, then take steps to make it as annoying as possible to simply download the GIF.

Tenor is owned by Google.

I had no idea. That explains a lot

It’s a bit ridiculous, really. I actually don’t feel the need to ban that many sites. I would probably be happy if I could just ban them. How can one site be so bad?

I’d like a ban lis option with public ban lists that I can elect to use so I can avoid having to spend time doing this manually.

Think of it as server side uBlock for search results.

I don't actually want Google to index as much of the internet as it possibly can. I actually want streamlined indices that are tailored towards thoroughly searching a specific knowledge domain.

Edit: I would love, love, love a search engine specifically tailored towards programming questions. One that doesn't treat queries like "C# ?." as me looking for "C#" and getting the most generic results possible.

Yes, instead of Google collecting information about me to improve my search experience, I want:

- A box where I can select the context of my search.

- The box then appends a specific string to the query.

For example, if I set the box to "clothes", the box will append "waist 34 inch, 5 feet tall" to the query.

If I set the box to "programming", the box will append "favorite_languages:Python,C++" to the query.

And of course a search engine should interpret it accordingly. No need to collect and store my personal information.

I just want a search engine that gives me AND, OR, and NOT qualifiers or something similar. Parens, too, if possible. Ideally, an sql interface, but that's probably pushing it :)

> Ideally, an sql interface

You could use Google BigQuery to write such queries for the whole internet (or parts you chose)

You can have that with my idea. Just leave the Combobox empty.

I had an idea for this where I was building a manually-constructed index of sites kind of like old-school Yahoo (related to typography), and then there would be a full-text search of those URLs and their close relatives (pretty much, the idea being that for a given URL, I'd index anything else in the same URL directory or subdirectories of that URL). I ended up giving up on it largely because time is finite and I have plenty of other projects competing for my time. I imagine something that built on the basic idea though could work, by omitting the Yahoo-style directory and instead working from a collection of vetted URL starting points.

If you have a (very) short list you can make/modify a search engine entry to add a -site:which.ever for each that cause problems. The problem is that the scummy sites are legion but the negate list / search term limit usually cannot handle many - I don't know what Brave's limits might be.

seriously, any Googlers around that are close enough to search to know why Pinterest is still dominating image results despite being incredibly low quality? this is a widely known problem.

We have had a few posts about this, a blog from an ex-employee explained it basically as Pintrest spending tens of millions to do everthing they can to game Google search results and they arn't going to stop any time soon. Pinterest just puts A LOT of time and money into SEO.

I guess that makes sense... but it seems like they should be manually penalized. They're single-handedly degrading the quality of the entire image search product.

pinterest the most useless website in the search results

yelp, pinterest, w3schools

Prior to mdn w3schools was a decent way to quickly find info on random HTML tags. Not sure why it gets so much hate.

Prior to the popularity of MDN and campaigns like W3Fools (q.v.), the articles on W3Schools were very bad from a web standards perspective (<marquee> and VBScript levels of bad); from a programming perspective, their PHP articles (which largely date from that time) are still incredibly shoddy. Their unwillingness to relinquish the unwarranted authoritative tone or at least visibly disavow any connection to the W3C (who asked repeatedly), coupled with their domination of search results, at a time of a standards push (remember validator buttons?) and multiple high-quality documentation efforts (later folded into MDN; Opera in particular had a good course) just plain got on people’s nerves.

They’re a valid reference, just a low-grade one, and using a low-grade reference is rarely a good tradeoff. They also used to be worse and never quite shook of their reputation.

For me the issue with all of these sites is that their primary innovation seems to be dominating search results, and everything else is an afterthought. I actually don't know very much about w3schools other than that I have to continually reach over them to get what I'm after.

I also seem to remember reading a lot of critiques like this one: https://meta.stackoverflow.com/a/280484

w3schools has its place but it’s usually the top result when I’m never looking for it.

Once it was useful. But now I memorized all the information there so it’s not interesting.

All the various gif sites that make it nearly impossible to get an actual gif. Why the fuck are they showing up on image results if the thing they serve to actual humans is a video? I thought tricking the Google Bot was supposed to be bad for SEO, but I guess not.

At least move it in the "Image" tab; (facepalm)

I can’t remember if it was a former feature of Google search or a Chrome extension, but that capability used to be available in Chrome. w3schools was on that list, among other republishers and whatnot. Loved it. Maybe someone else can augment my hazy recollection.

You're probably thinking of "Personal Blocklist", an official Chrome extension developed by Google. It's no longer available.

https://www.ghacks.net/2010/03/18/blacklist-google-search-re...

Someone should inform https://killedbygoogle.com/ to update their site.

That actually sounds awesome. But the user in control? No wonder Google killed it ...

That’s the one! Thank you.

Yes! I swear I remember banning some of those spam domains in 2010 or so. IIRC, I thought it was part of Google search.

It definitely was. Sadly it was short-lived, it lasted maybe a year. I remember banning experts-exchange and being a lot happier.

> Sadly it was short-lived

Understandable since a few years later Google would acquire Doubleclick Inc, which back then was number one in polluting the web with advertising, and first entry in all adblockers kill lists.

They sure want to improve their own user experience. The "use your information to make tons of money" experience.

I just found a timely example of one of these sites that republish GitHub issues as their own content. Trying to understand a fairly specific Tensorflow error. Pasted an exact phrase query and the top result is gitmemory dot com. The content is horribly formatted, obviously just a low-effort scrape job. The author is attributed, but the link links to another page on this site that even says it just pull the data straight from GitHub's API. The footer links to another website, uonfu dot com. It looks even worse, and doesn't even try to attribute the original author of the content.

https://search.brave.com/search?q=%22%27tensorflow.python.fr...

> If you are wondering where the data of this site comes from, please visit https://api.github.com/users/.../events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

Better reading experience, my ass.

Yes, I hate it when I get results for that site. It also squeezes out the original github.com URL for that content from the search results, as you said, so I can't even solve it by writing an extension to hide it from search results.

Luckily the original URL is easy to recover from its URL ( https://github.com/tensorflow/tensorflow/issues/26922 in your case), so I just copy the URL from the search result and fix it up manually.

On the other hand, I actually like the fact that such mirror sites exist, because they often have content that's disappeared from the original for whatever reason. The name GitMemory is itself suggestive of that.

Mirrors can exist without gaming search to supplant canonical sources

Worst shitsite on the web. It contains nothing useful, original or unique. I'd like to know what's Google's excuse for blacklisting that domain into oblivion.

I have a greasemonkey script that blocks domains from being shown in GSE results. Works well, but I don't have it on all of my browsers.

I remember reading in HN an idea about allowing people to customize the weights used to filter search results - a search engine that provided that, where I could use some curated configuration file. Maybe I could have profiles like "cooking", "programming", "news" and help the search engine even better to understand my intent.

Or just have the "don't show results from this site again" button from youtube taken into google results, and allow me a quick access to that list to manage it afterwards - ALSO, allow me to add a comment to remind myself WHY I did not want to see it anymore.

This is where ad-based businesses show their ugly side: The more control you have over what you see, the fewer ways they have to slip you inconspicuous ads.

This is why all content feeds like Facebook and Twitter get vaguer as time goes by, and it's why Google will never give up control over results unless it legally has to.

Yeah. The dependency to adds, are a big reason why so much tech suck so much, despite trillions invested.

Because those investments are not made, to show you clearly what you want to see, but to show you just enough, to not loose you while injection as much information garbage as possible. If we do not fix that, it will all just get worse, not better.

It's also why every social network moved to an algorithmic feed. Had it remained chronological, our primitive brains would have noticed content from yesterday come up after a certain point, signaling to us that we are caught up and don't need to scroll further.

Can't imagine how catastrophic that would be to their ad business.

G*d forbid anyone ever stop scrolling. I was browsing airbnb the other day, looking at places in a small area (maybe 25 rentals) and was shocked/alarmed/saddened to discover that the app has infinite scrolling. Rather than the list ending once I’ve seen all 25 results, they just show them again in a different order.

That's both garbage and outlandish design...

Not to mention the fact they police content. They'll refuse to associate with anyone they deem objectionable enough for any reason. This will deny sites their revenue and will lead to self-censorship to regain their favor. Nothing kills perfectly good websites faster than some offended person complaining and ads getting pulled.

If you control what articles and pages show up based on some searches you can impact elections. Forget ads, who would give up control every close election on the planet?

There's a Chromium extension, uBlacklist, that does something similar. You can blacklist sites from search results and also whitelist so those are highlighted.

Excellent! I was looking for precisely this. Next, I would also very much like to weed out image/video/map suggestions, related searches, people also ask and page snippets. The cognitive overload from navigating these giant banners and getting to my search results is getting worse by the day.

But, how stable is this? I tried so hard doing this with greasemonkey. But google's DOM is ridiculously obfuscated and kept breaking frequently. So much for the semantic web, heh?

https://github.com/iorate/uBlacklist It's on Firefox as well as Chrome, and apple stuff.

It's like a gift from god that allows me to block pinterest image results. Not had any issues with it what so ever.

Need to try this out!

Brave put out a white paper regarding a version of this.

Have they followed up with a product/feature?

It was the only portion of brave’s approach that was interesting to me. It’d be disappointing if they dropped it.

Brave Goggles: https://brave.com/wp-content/uploads/2021/03/goggles.pdf

This is actually doable. There's a tweak to PageRank you can do (that's as old as the original paper[1]) that allows you to bias the ranking toward a certain set of websites. It works really well.

While it's probably unfeasible (or at least really expensive) to do completely personalized rankings, that's just too much data, but segmenting off into areas like academia, blogosphere, tech, etc. is quite doable, and as the authors remark, this approach is highly resilient to manipulation from commercial interests.

[1] http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf

> Or just have the "don't show results from this site again" button from youtube taken into google results,

millionshort.com has a "block this domain" option. It can be useful to combine it with the filter "hide results from top 1,000 sites".

Searx can make you do exactly that. It's really easily to get started with a self hosted serverand then you can customize it the way you want.

Interesting, I did not know it could do that - need to take a look into it!

I use Neeva.com and I think one of their features resembles this idea. It is a paid for search engine though.

Neeva has been pretty good in my experience, but the lack of ability to make it the default search engine in Safari (macOS and iOS) is what prevents me from using it 100%.

That's obviously not at all a knock against Neeva. Safari needs to allow more flexibility in its search engine defaults. I've filed multiple radars/requests to Apple to make this setting more flexible and/or provide Neeva as an option.

I think neeva is interesting. Once payment and a login is required it changes the incentives and opens up new feature possibilities. One cool feature is linking your private cloud document storage making search very personal.

It remains to be seen if they can get enough people to pay in order to keep going though.

They won't let me try it because they "haven't launched in my region". Presumably that's because I'm in latin america, and they're conflating location and language.

Hmm, a paid search engine - wonder how they handle the spam SEO problem, but is an interesting idea

I'm not sure how they actually do it, but it works pretty well. When I search for things I almost always find what I'm looking for. I use it both for tech and for personal stuff.

Kagi Search has "lenses". I have a "Programming", "News 360" lens (news from all sides), HN lens, PDF/Whitepaper lens...

You can see how it works here:

https://twitter.com/vladquant/status/1445298385800470529

(disclaimer: Kagi founder)

https://felvin.com/ is trying to build "Search profiles", but it's still quite early.

Thanks for sharing, gotta look into this!

Yahoo had a system along time ago where you could slide results more toward shopping or more toward informational. Like most features like that, it didn't last very long.

>I really wish a search engine would empower me to provide a URL ban list

Brave discussed implementing similar concept called Goggles:

https://brave.com/wp-content/uploads/2021/03/goggles.pdf

Kagi Search already has customizable 'prefer' and 'mute' lists for domains, as well as customizable 'lenses' which are similar to goggles concept from Brave. (disclaimer: founder at Kagi)

https://kagi.com

I'm not even sure how much manually blocking domains would help remedy the problem with Google results, which is that so much of the top hits are copy-pasted blog spam with very little information, scrambled together (very possibly by a non-human) purely to game SEO.

It seems like it's gotten even worse just in the past few months. I'll try to Google something common, like "can you feed X to dogs" and all the results I find are these giant "articles" that aren't even about that specific thing. Instead it's a giant wall of text with commonly-Googled questions, and if you ctrl+F to the section you were looking for, the answer is usually horrible, and why should you be trusting information from this website anyway?

Then you go back and click through other results and find an entirely different website with all the exact same text on it.

Top recipe results are also all copy-pasted SEO spam surrounded by a wall of text about the history of the recipe, and how the author is a "country mom" (definitely not a man at a content farm in India), hiding the recipe deep within.

They must be pumping out all these fake dedicated websites at such a high rate that blocking domains won't get you anywhere.

What we do (at Kagi Search) to address this problem is use our own index which contains only non-commercial results from the web as well as forum discussions. Thinking behind is that the quality of the content is in inverse proportion to the number of ads/trackers/affiliate links on the site.

Our index is still in infancy but the example of it working for the query 'best laptop' can be seen in this screenshot:

https://imgur.com/ypyOilV

> Thinking behind is that the quality of the content is in inverse proportion to the number of ads/trackers/affiliate links on the site.

search.marginalia.nu does this and for the stuff that it can find it works wonderfully.

Getting results on search.marginalia.nu is (borderline) delightful in the best cases. Last month I searched for something along the lines of "dual boot windows linux" and got 3 fantastic results among the top ten - and no blogspam.

If SEO specialists figure out and start to reduce ads, trackers and scripts generally I'll count that as a win too :-)

afaik, seo spammers are quite sophisticated in masking affiliate links and tracking codes with redirect chains and other techniques. Their general assumption is that google does exactly what you expect to do and downranks affiliate spam.

What are your thoughts on reputable news websites that have tons of ads and trackers because they have yet to figure out any other viable way to survive?

I think that 'reputable' and 'tons of ads and trackers' do not belong in the same sentence.

Ad-supported business models incentivize the creation of large quantities of content, because you need a lot of pageviews to earn a little bit of money with ads (since most people either ignore or block ads, if they manage to load the page to see them at all).

High quality journalism should have value that we are ready to pay for, like we did for hundreds of years. This concept is called the "newspaper".

You are expected to pay for high quality baker or a tailor, book, movie or music, why not for getting information that "only" has the power to shape societies? If I see information next to an ad, I personally tend not take it too seriously.

What is missing right now from a technology standpoint is an easy way to manage subscriptions, built into my browser as default. Expecting the user to create and manage a separate account/billing identity for every publisher is what is preventing this model to take off (IMO).

And people already complain about paywalls. And even make tools to skip past the paywalls.

The problem with paywalls:

1) they ask for an unreasonable amount of money. If they could just charge what they'd otherwise earn from ad impressions on that particular page view, it would be fine.

2) there is no global, anonymous (from the website's perspective - I don't mind the payment processor keeping records for AML/KYC purposes) micropayments system, and card payment fees make micropayments unsustainable

3) subscribing to the website requires providing personal details, with no guarantee they won't be used for tracking/marketing/etc. Cancelling a subscription is also intentionally made difficult - see the New York Times.

4) all paywalls require subscriptions - there's no "pay per view" mechanism. Do they really expect every web user to have a subscription to dozens of different news websites? Unless you literally spend all day reading news, it's bad value for money.

Sure it would help. Adblockers have had custom filter lists for years. If all you need to do is upload a text file every once in a while, that would go a long way to improving search results in general.

I didn't click into the paper, but I see now it also mentions crowdsourced filters. That would be much more effective than everyone creating their own blocklists

I have personally found that when searching for developer documentation, as you mentioned, unfortunately nothing beats Google. I can always find what I’m looking for very quickly. But when I try DuckDuckGo, it’s a mess of irrelevant or weak pages.

I also noticed that, if DDG doesn't find it in my initial search, there's no way on earth I could game the search terms to actually find it. So if my first 1-2 searches give nothing I just add the !g...

I’ve learned just cut out the entire exploratory first couple of steps and just always use !g when searching for anything developer related.

Same thing with bing. It's a fine search engine for lots of things, but for searching documentation it's really weak, showing sites like w3schools and tutorialspoint far above the results I actually want.

It kind of makes sense that w3schools would spend much more time on the SEO game than docs.python.org, but it just drives me back to google

I noticed the exact same thing there. Usually, when I don't get results I just append !g after the search terms, and google yields much better results. But what I'm most bothered by DDG nowadays is that on my phone the first 2-3 results (which on a phone basically means the whole visible area) are quite frequently ads.

It’s actually possible to switch off ads in the DuckDuckGo search settings: https://duckduckgo.com/settings

Awesome, thanks for the tip!

It doesn't need to be better than Google, it just need to be better for its marketshare. If users are mostly dev, than it's a win.

Some years ago Google had the option to remove a website from search results. After a few months they discontinued. It would be nice to know why, if the user adoption was too low, the computational cost too high, or the user experience too bad because users inadvertently removed good results.

Like most features that Google releases, I believe that they use them to train their algorithms and once they get enough human input on a feature from people using it, they remove the feature and turn it over to the machine. We see it on the SEO side all the time. Release a feature, call it a ranking factor, thousands of SEOs jump all over it. The algo learns and then the features importance is removed. NoFollow links are a great example of this.

Commercial pressure seems the most likely to me

How would such "pressure" look in practice? Seems to me that every industry on the planet is at Google's mercy, not the other way round.

How many google ads are on github or stack overflow compared to the other pages you find? They know exactly what every little change to the algorithm does to their bottom line.

Search doesn't have to concern itself with AdSense ads, search ads still make up something like %85 of Alphabet's total revenue

The worst sites a.k.a. those a user might want to block are probably also bringing Google disproportionally large amount of income is my guess.

I'd put my money on performance costs.

Personal remove lists basically turn each query from each person with such a list into a completely distinct query, which breaks caching on multiple levels. If a few people are using such a feature or if people sometimes add a couple -sites to a query, no big deal, but if enough people used it with basically unique site lists, the performance degradation would probably make a rollback of the feature inevitable.

I can imagine it was also a handy tool to DoS a site.

It was a tool to remove a site from your own results. If you searched and saw a site you didn't like, you could click to no longer get the results from that site. It wouldn't affect what anyone else saw anywhere.

That's a good point. I guess bad actors (exactly the same ones that publish those shit-sites) can abuse it to block the actual sites.

Not a problem as long as it only affects the user themself?

My guess is some ux head exploded when they realized someone might blacklist a site they wanted. Or maybe the one who made it got promoted and no one stepped up to maintain it?

Adding a minus sign in front of a site: specifier still works for me (eg "google -site:google.com"). Works on DDG, too.

DDG is awful for spam on most searches which can't be easily resolved with a Wikipedia info box. The 3rd result onward is very fishy, content spammy results of very low quality. It's like their algo isn't good enough to separate the wheat from the chaff yet.

DDG can give downright weird and irrelevant results, and sometimes (rarely) even returning with "noting matched your search". I hate it, but I have to fall back to Google quite often, and the results are consistently better; sometimes much better.

I still have DDG as the default because it does work a lot of the time, and because it's easy to turn a search in to a Google search with !g, but I have to admit it's not all that good in comparison :-( To be honest I'm not sure if I would keep it as the default if it didn't have !g, and that's not a good look for DDG :-/

(I appreciate this is a very hard problem btw, so not even intended as a dig at DDG; just my experience with it)

I hear this criticism a lot and I always wonder whether that has something to do with either me not having a Google account and thus using the unpersonalized search, or simply with what I search for, but Google is almost always giving me worse results that DDG or, for that matter, any of the smaller competitors.

Google seems to be filled with sponsored results that are only superficially relevant, and from page 2 onwards, it sometimes feels like whatever it’s giving me isn’t related to my search at all. Even when I’m very specific with my query, using +/- and quoted phrases.

I'm never logged in to Google, and don't even store cookies by default (although that may or may not be enough to prevent all tracking/personalisation, it's more than "the average" user and does seem to keep the worst out, not that personalisation is necessarily bad; the singular reason I have an account is because I actually like YouTube personalisation – without it I just get idiotic nonsense on the frontpage and now I get stuff I actually want to watch, although the old YouTube model of good categories was still better, but ah well).

I wish I had taken some screenshots, here's the only recent one I have: https://i.imgur.com/b9hu2a9.png – I was trying to find about Cambodian date writing conventions for some i18n code, but all DDG gave me was spam (and most likely, scam) results. Probably not the best search term in the first place ("Cambodian calender" would be better), but it's a decent example.

I should keep a spreadsheet or something, but I'm too lazy for that. And it's not really all that interesting either.

Meanwhile I was trying to find a romantic partner in Cambodia and Google kept giving me useless calendar-related results.

Sure no Natural Intelligence nor an algorithm could've possibly guessed what you meant with 'date in Cambodia'. Reminds me of the time Blanche Deveraux remarked on the obvious nature of a book titled "Females to Fondle" only to find out it was a tome of the Encyclopedia.

I'm not sure if I did it right, but I searched for "Cambodian date writing conventions" and got this as first result.

https://en.wikipedia.org/wiki/Date_format_by_country

Yes, my search terms weren't the best, and I did find what I wanted soon afterwards. However, just the inclusion of "dates" shouldn't bring up loads of spam (and most likely scam) sites.

(What I actually wanted to know is how widespread usage of the Khmer calendar is, and if I could get away with just supporting Gregorian).

Same for me... I think maybe part of it is that people have forgotten (or never knew) how to choose good search terms, and Google has worked hard on giving good-for-most-people or good-for-your-profile-bubble search results for terribly-constructed search terms, while the others haven't. I've long had the habit of constructing search terms that are minimal but distinct, since I had to do a lot of searching on less-DWIM systems like library catalogs and journal indices dating back to the 90s, and this seems to serve me well with searching DDG. I expect Google simply doesn't care about your actual search terms very much, and gives you stuff matching some kind of linguistic model of your search; what it thinks you want to search for, rather than what you ask for. That would account for it giving worse results on a well-constructed query.

Here's another one I had today: https://duckduckgo.com/?t=ffab&q=maan+leo&ia=web

Maan Leo is a Dutch author. DDG gives me a few bad results about Maan Leo, a few bullshit links about horoscopes ("Make a Leo man chase you"), and finally the Wikipedia page for Low Earth Orbit (LEO).

The Google search gives me results about, well, Maan Leo (in spite of using Indonesian search by the way – this GeoIP stuff Google does is so annoying). For this particular term, DDG is basically useless.

The Brave search results also seem a lot better by the way. Certainly loads better than DDG.

I have heard a few people say things like this, but I have been using ddg for years and I think I could count the number of times I got nothing back on the fingers of one hand, and those were times I was using incredibly specific search terms +"leprachaun eating marrow" +gif -funny or whatever (haven't tried that exact search but you get the idea).

Occasionally I try google search to check and I virtually never think google's results were better. But I'm guessing a lot of that is down to how you do search-fu, what you're actually looking for and subjective stuff about ranking quality.

Come on, "dates in Cambodia"? How do you search for local trendy music, "Hot singles in my area"?

Isn't that a bit by force of habit ? I used to fallback to the !g quite often. I hardly needed it in the last 2 or 3 years. And Google was consistently worse when I did. So, as you say, just my experience.

DDG also just uses the bing search API for all their results seemingly. They claim to use other sources as well, but in the tests I conducted building Kagi last year, it was always identical to bing. They must have a special deal with microsoft because the API fees at that volume are completely unsustainable, as in the operating costs are 5-10x the theoretical profits from advertising. If they were actually scraping, this wouldn't be a thing: https://help.duckduckgo.com/duckduckgo-help-pages/company/ad...

Yeah, DDG is just privacy Chrome on top of bing. As far as I can tell they've made no efforts at building a proper scraper or index. If Microsoft gets tired of the deal there is no more DDG. Risky proposition.

It's also horrible for non English content from my experience.

I don't understand why search engines don't fight SEO spam using simple, hardcoded rules for specific keywords. Surely, something like "if the search string contains the word 'python', rank results from 'python.org' above everything else" can't be difficult to implement?

Granted, it's not a silver bullet for search engine spam in general, but even a few dozen rules like that would dramatically improve the quality of search results for a huge number of queries in practice.

I hate it to say it, but for CS beginners python.org isn't a good resource: it's simply too technical. Even for motivated students it will take them at least a few months of learning from a more beginner-friendly resource before the official docs on python.org make sense.

That's not to mention the multitude of other materials on python.org other than the docs, like PEPs, pypi (third party libraries), bug reports, etc. Can you imagine a student searching for how to use a Python feature, but found a PEP illustrating the design of that language feature?

> Can you imagine a student searching for how to use a Python feature, but found a PEP illustrating the design of that language feature?

That wouldn‘t be so bad. PEPs contain a short summary, motivation, and usage examples.

That said, I do get your point.

What about people who keep snakes as pets?

Their queries (such as "best food for reticulated python") are unlikely to have results from "python.org", so such rules would not affect them.

There's room for ambiguity. python egg, python length, python container, python types, python size, python lifespan

"For security, it is best to manage Python within its own dedicated container. Live Python in an unmanaged environment is unwise and can lead to resource strangulation. When objects are consumed by Python, this will affect their lifespan."

I think you’re broadly talking about site authority which is something search engines absolutely do, I imagine it’s a non trivial thing to get right and the scope of doing it manually for every topic on the planet is probably not going to be easy.

Haven’t used Brave Search yet, but I have not had the same issue it’s DDG. I use it as my primary search and it works the large majority of the time. I do resort to Google on rare occasions and am always amazed by all the blatant spam at the top of its results.

> I continued to be baffled why such blatant spam can consistently make it into top results

Same here. It's like they don't even care anymore. The results are just there to hold your interest so they can show some keyword ads along with them.

I wish people would start maintaining curated web directories again.

Google clearly gave up on fighting spam some time around '08 or '09.

I think they realized it's expensive, and not actually good for business if the spammy stuff is using your own ads and/or analytics anyway.

This is why almost all of the web looks like a shady spam-site if you don't have Adblock enabled. Google doesn't give a shit anymore.

Seems that way. Yet another reason not to feel any guilt when using uBlock Origin.

(tangent) It's hard to think of a website that wouldn't benefit from ban lists of some kind. Twitter's muted words feature is the predominant feature keeping me on the site.

I wish something similar existed for this site, Reddit [1], even news sites like NYT (at some point I've listened to the POV of certain contrarian columnists and don't need to see their byline anymore).

[1] I know you can do this via Reddit Enhancement Suite but it would be much better as a native feature that worked across clients

Reddit is effectively used as a covert propaganda and advertising machine so its absolutely not in their interests to provide that functionality sadly.

Well, I'd bet a big factor is that these spammy sites probably spend a lot of time on SEO - illuminating StackOverflow answers and official documentation don't.

I guess I’m surprised that it’s not possible to quickly deindex these spam farms and render their SEO spend a waste. Especially Google, I’m surprised they can’t detect and ban SEO rings more effectively.

I am surprised by this as well since Google is able to identify the origin of content (eg. Stackoverflow) and penalize sites for duplicating that content (or so they say). In my opinion it wouldn’t be difficult to identify the domains duplicating the content and penalize heavily. But, those sites are most likely displaying Google Ads so…

I would assume this is easier if you are indexing more frequently and have been indexing for longer, and Google has more resources to allow them to crawl more often (another advantage to scale in search, which combined with other advantages of scale, is why Google has an effective monopoly IMO).

If you recently started crawling you won't be able to detect which content was 'first', and if you don't crawl as frequently as google you are reliant on your crawl frequency being quicker than the 'content stealing' speed.

> if you don't crawl as frequently as google you are reliant on your crawl frequency being quicker than the 'content stealing' speed.

No problem there. If a page hasn't been crawled, it won't be a search result.

I mean you need to crawl often enough that you can determine which site stole content.

If a full crawl of the web takes 3 months and A steals content from B after 2 weeks, it may not be possible to easily infer that A has stolen from B rather than visa-versa.

If I crawl every week and see that A posted it on the first crawl, and B suddenly also had that content on the second crawl, I can infer that B stole from A.

I just wish I could configure google to not show certain domains such as W3schools without adding a complex search query or installing plugins

To be fair, w3schools and the like don't just copy content. They provide different information and, most importantly, present it differently from official docs. And for many, their way of presenting information may work better.

I acknowledge your opinion but still do not want to see that and several other spam sites over more official sources such as mdn.

The W3schools hate is a bit outdated. Even W3Fools[1] eventually backtracked and acknowledged that W3Schools has continuously improved and updated their content.

1. https://www.w3fools.com/

This is such an easy optimization that Google has to have tried it and seen their ad revenue fall.

"Permanently remove this site from search results" seems like the absolute easiest "personalization" to implement.

And the most certain too. You don't need to guess whether or not the user really needs that result

But what will Google do if users end up banning the majority of websites with Google Ads and/or analytics?

The reason Google doesn't fight SEO spam (and just generally annoying/obnoxious behavior such as paywalls) more aggressively is because it doesn't want to bite the hand that feeds - all these spammy sites have ads or analytics that benefit Google.

Maybe the good folks at Apple, Mozilla, Brave or Opera could implement this feature in their browsers? They’re looking to increase their percent of browser use - it could encourage people to at least try switching over. Obviously would be popular among the HN crowd for developer documentation, but we could spread the news :)

On the downside I could see this being used by terrible people to delist actual information sources so that they would only ever be served conspiracy theory sites in their searches as well.

I don't think the first two companies you've mentioned care about making search better, as they're both getting a fat check from Google.

Google did it with a lot of health blog spam that had low quality and false info but a ton of referral links. It takes a lot of bad publicity for them to act, though.

The SEO places aren't dumb, they do their best to mimic legit content.

True. If you search for any Python documentation the first links on Google are almost always geeksforgeeks, w3schools, etc instead of the official Python docs.

I think this is telling of how often the python docs actually solve peoples’ questions. For example, I searched “python string replace”, with the first 4 being tutorialpoint, w3schools, geeksforgeeks, stack overflow and at position 8 is the python docs.

The issue with the python docs for this search is that it takes me to “built-in types”, and starts off with things totally irrelevant to what I want. I now have to go searching for string, then find the built-in method replace. Meanwhile, the other tutorial-esque websites almost all show an example of str.replace above the fold. If you’re Google and you’re trying to get people information faster, you’re going to promote the result that has more people not coming back to the search result page afterwards because it was hard to find the answer on that result.

I think string replacement in Python is a particularly egregious example because strings are core language types. Library docs are a lot better organized and a lot more easily indexed. This is just a problem with Python's docs in general, imo.

You want Google to direct you to the correct anchor on the page. That makes sense, Google should do that. The docs shouldn't be broken out to 1 page per method because Google likes to link to the top of the page instead of the relevant anchor tag.

This is true, and if you want the docs you can always just add the word docs to your query. Expecting the search engine to always favor one thing seems like it'd work well in this case for some people, but in other cases might get mushy

I personally configure custom search engines for things like this so if I want docs specifically, I'll type something like "py collections" which directs me to https://docs.python.org/3.10/search.html?q=collections.

I find this especially useful since I often otherwise find myself typing site:python.org (or similar things, like if I just want to see the MDN page instead of going through w3schools or blog posts for examples).

edit: in the particular example for str.replace, I found it buried in the results for "py replace", at the top of "py str.replace" (helps if you know what Python calls the type) and basically nowhere for "py string replace".

Yup, python docs need to look at cppreference

You should try adding "docs" to the query. "range python" shows w3schools at the top for me but "range python docs" shows docs.python.org first for me. Same for "html article tag" (w3schools) vs "html article tag docs" (MDN).

I filled in the beta signup form for Kagi search [0] the other day and one of the questions was along the lines of "What must-have feature do you need in a new search engine?", and my answer was the ability to blacklist these spam sites for the exact same reasons. The major search engines are becoming close to useless.

[0] https://kagi.com

my answer was to rank sites higher the less ads they have

This is really bad on Google too. I fixed it by installing an extension that lets me hide certain domains from results

This used to be a built in feature. Being able to block ExpertsExchange from all searches was incredibly useful. Now I'd do the same for Pinterest, W3Schools and apidock.com

I wonder maybe every other day if I could delist W3Schools, by maybe always adding a negating query aspect to the URL. But this all doesn't seem easily supported, when I looked at it. Yep, hating on W3Schools with a passion. Younger self shot himself too often in foot with W3Schools.

Warning: unusual opinions sliding into a full fledged rant.

I know it is cool to hate w3schools and I don't use it personally, but seriously, can we get over ourselves here?

The reason why w3schools rank highly and have done for years is because for a large segment of users it works.

And as someone else has pointed out the official documentation is often atrocious: both Python and Java online docs are close to worthless if you have a decent ide that shows you method signatures.

PHP docs used to be a lot better in that they described what was going on and why and had comment sections that I imagine they also used as feedback to improve the docs.

I haven't used PHP or Python for years but today there is Spring docs to drive me mad: telling me all the things I know and not telling me anything about how to interpret a Spring project.

Spring is infinitely flexible and there are more ways to configure Spring than there are Spring programmers since most of them can't even configure two projects the same way ;-)

Not in regards to W3Schools, but in regards to docs in general: i've found those of PHP to be excellent for one simple reason (regardless of what i think of the language).

They have user comments, which oftentimes contain the information and examples that are actually very important in day to day usage, as opposed to a cut and dry description by the language authors.

For example, go to this page and scroll down to "User Contributed Notes": https://www.php.net/manual/en/function.str-replace.php

To me, it feels like most of the technologies out there could benefit from user source and user voted content at the bottom of pages like this, the one thing that IDEs also don't provide (though if one were to throw in some fuzzy search from, say, StackOverflow, we could probably get context surrounding a particular bit of code).

Of all the spammy sites, w3schools is the least of my problems. I still find a use out of it when I need some very superficial information about, say, an HTML tag.

My hate for W3Schools comes from the low quality of the content and the impression I get that their high ranking is only the result of SEO hacking. I respect the fact that there's a lot of _poor_ official documentation. But let's not supplant that with different _poor_ documentation.

The content is not low quality. It used to be pretty bad like 10 years ago but no more. I prefer W3Schools to MDN.

I find W3Schools fine for most use cases. What would you suggest as being a better resource?

I can't speak for Java's docs, but Python's have good examples in the documentation that are not present in an IDE's tooltips. Further, there's often a handy link to the source code of the module.

Thinking back, yes it is probably true what you write about Python docs, the problem is you easily end up in the wrong docs when searching for something.

I'm not sure why that's happening to you. Perhaps you're being misled by some SEO to click on pages that are sites trying to put advertisements in front of you?

Not the same but I admit my explanation might not be the best.

See https://news.ycombinator.com/item?id=28927092 for a more descriptive example.

You don't have to use W3Schools (I prefer MDN myself) but you can forgive them. They've gotten better over time: https://www.w3fools.com/

Just add the following in your Google search box: -w3schools.com

I meant doing so without adding it to every search. But, hey, I clearly didn't have enough faith in humanity... not only am I now able to find some extensions to generally mess with my search queries, there's extensions freakin' DEDICATED to filtering out W3Schools!!!

https://addons.mozilla.org/en-US/firefox/search/?q=w3schools

(Edit: that doesn't mean I necessarily trust or recommend niche extensions like these.)

>I really wish a search engine would empower me to provide a URL ban list

With Google, you can append -site:site1.com -site:site2.com to a query, though I don't know if it provides the same result as filtering them out after a query.

I doubt it would do much good to have a ban list. An entire industry is singularly focused on gaming search engines to promote whatever. It just so happens that blogspam sites are incredibly profitable in aggregate.

It still seems like such a simple technical problem to solve, albeit at large scale. Particular URL is blatant spam? Delist it. Keeps popping up on new URLs? Threaten to delist the hosting provider's address block. I'm more inclined to think that the reason these problems aren't solved is because there's little incentive to do it. The more time you spend milling around looking for what you're looking for, the more opportunities there are to show you ads.

Right, ad-driven search engines are fundamentally incentivized to provide results that are bad enough that you need to try harder to find your answer, but that are good enough for you not to switch (with a fairly high cognitive barrier to switching).

I would pay for a search engine that let me 1 click ban results from a given domain from showing up. I never want to see reddit, facebook, twitter or pinterest in my results.

Interesting. I find reddit results are some of the most useful results but maybe that's just related to my specific hobbies/searches.

> I’m having troubling telling whether it’s just my own halo effect or if the initial quality that experienced has started to slip a little

I've had the exact same experience and sharing the same doubts as well.

For programming related queries I'm going to Google 100% though, it understands the intent so much better. But for general queries I use Brave 90%+ of the time. But, it feels like I'm adding !g to the query more often than I used to in the beginning.

That's an interesting observation and one I think rings true even though I haven't really actively noticed it. Stuff like tutorialspoint or w3schools is fine imo since they actually write their own content and add a but of beginner friendly explanation to it, but there's some blatant stack overflow copying that's been popping up.

At least they have the option to report bad search results so they might be able to improve it in the future again.

Does it have a page two yet. When Brave search was launched and discussed on HN Brave search only had one page of search results. It was like five or ten results.

How would you say that the Brave Browser and Brave Search compare with something like Firefox and DuckDuckGo (plus a plugin, like uBlock Origins), if you've used those?

Anyone else care to comment? Personally, i'm still someone that's used more to Firefox and is more familiar with Mozilla due to them having been around for longer, but their recent decisions might make some people reconsider their choices and look into exploring new options.

For browser biggest UI difference is tabs - they squeeze instead of scrolling. There are no plugins that can create a proper sidebar, but out of the box you can organise them into foldable groups.

Search results are noticeably better, especially for non-english and location-specific results. What's worse is image search - unlike with text you tend to get only 2-3 good, relevant results, everything else is only somewhat related. I suggest giving it a try even if you'll decide against browser switch.

Stock Chrome has a flag that activates a combined two-tab reading list + bookmarks sidebar. Brave has the same flag, but for some reason the sidebar didn't list bookmarks the last time I tried.

There are actually flags for enabling tab scrolling if you want, and tab squeeze is also adjustable via flags.

Thanks, this really helped with a major pain point I had with tab management in Brave.

UI wise the big thing for me are tab groups. They are straight crack, and the stock Chromium implementation Brave inherits probably my favourite.

Apart from that, the main thing you can't adjust via flags is Chromium's style of using separate browser profiles instead of Firefox's multiple containers within one profile/window/session.

The bigger UI differences are on mobile, where again, tab groups are crack and the implementation excellent.

> I really wish a search engine would empower me to provide a URL ban list that gets applied server-side instead of filtering on the front end (if anything).

neeva.com could do that for you for the price of a ~coffee ($4.95/month): https://www.youtube.com/watch?v=cc2aTB24XmI

I joined their waitlist and I eagerly await. If there's any kind of invitation scheme that speeds that process up, I'd be ever grateful.

I'm getting these sites with Google too, so I don't think it's just a Brave Search issue.

Looking at the settings, it looks like everyone is opted-in the data collection by default, and you have to manually opt-out, which is a nightmare for people like me who delete their browsing data on browser exit. I will stick with DuckDuckGo.

I completely agree. I almost guarantee if they hired a couple hundred people to look for spam and isolate that out and feed it back into ML models they could eliminate 80% of it. It's SO obvious.

Just let me mark my results (and let me bookmark them) then let me search in that subset of bookmarks aswell as use them in your algo to improve general search.

> Talking about sites that just republish GitHub issue threads, republished StackOverflow questions, w3schools-likes, etc.

I have the same feeling, but using Google ;)

The spam google has for this is outrageous.