It's ironic that, in trying to make search "human friendly", Google has also succeeded in giving it all the negative traits of a human --- the "human" that is more like a commissioned salesperson.
I know the argument is that humans frequently make mistakes so "we should just show them what we think they wanted", but that's just opening the door for manipulation.
Search was much better when it was closer to a "grep the Internet". IMHO machines should remain "dumb" (for lack of a better adjective) and leave the important decisions to the users, keeping the latter in control.
Curation-as-a-service could even become a viable market. It's arguably already happening to a certain extent through various platforms, creators, and aggregators; but it's not really split off as its own service yet.
The dumber search is, the easier it is for the bad actors to game it and drown out the legit sources.
Maybe page rank (mentioned in a sibling response) helped address this, but then it’s really no longer just a dumb grep.
A simple "-site:somesite.com" removes those, or just scroll past them. Keyword spam is easy to ignore because the title of the page, the domain, or the path often has nothing to do with what you're searching for, and the preview text is nonsensical.
The dumber search is, the easier it is for the bad actors to game it and drown out the legit sources.
The dumber search is, the easier it is for users to ignore the crap and find what they're looking for, even if it's not the first result. The "smart" omission of results has the effect of removing a lot of useful ones too. Google's claims that they have X number of results for a search is essentially meaningless if you can't actually see them all.
If you really believe that, maybe it is a nice startup opportunity for you :)
So to share some of the challenges with that (if anyone out there wants to try again) they are as follows:
* 'Search' means different things to different people, and we've been trained that a 'search engine' finds anything on the web (for the most part). The product Blekko built could more accurately be called a 'reference' engine which was used successfully by people trying to find facts or data and were not generally trying to find things to buy.
* Have your own advertising system, all in house, where you don't have to "revenue share" with anyone if you don't want to. At its peak Blekko was serving over 10M queries per day which, if we had owned all the advertising revenue on those searches would have kept us going and growing. That said, building an advertising system is both difficult and fraught with patent risk / bad-actor risk.
* Don't let anonymous users use the service. This is perhaps the hardest thing, most people won't give up an email address for even the most useful of services, however since you're spending money serving up search queries you don't want to waste that money serving up queries to bots and other bad actors. At any given time when I went through the logs there were between 5% up to nearly 18% of the queries were 'suspicious' or likely bots. That is 18% of your capacity you can't give to "real" humans if you can't control that traffic.
* Build a relevancy ranking system rather than a popularity ranking system. Search results have two metrics of interest, precision and recall. For a reference engine you want to focus on precision over recall. And while existing search engines use the "virtuous cycle" of search & click to track popularity (which can be an indicator of precision but is better at indicating click-baityness) build your ranking engine using NLP based evaluations.
* Your document index size should target 5 billion documents with a goal of 10 billion documents. Scale your cluster and algorithms to process a query to that index in 100mS or less.
Do that and win :-) Or find the next barrier to creating a useful way to search the web for information.
Similar to your biking anecdote, I was trying to find any simple trouble shooting help for a home coffee brewer that suddenly stopped working. I couldn't find any results in the first 4-5 pages that weren't trying to sell me a new machine.
The manufacturer is also partly to blame, because I couldn't find anything remotely relevant either on the product page or in the PDF owner's manual.
It's a shame what these tools have become when they could be so much more.
I think one of the reasons is that I do not get a bunch of the ‘smart’ stuff that Google tries to do for me. In a way it’s like using a search engine from 10 years ago ... and it’s better.
There are things that I do miss — quick cards for things like currency conversion, flight details, that sort of thing - but the mighty Duck is getting better at those, and if I need to reach in to Google I can always just !g it.
The other infuriating thing that's related is if you're trying to find a schematic or service manual, Google thinks you're just looking for the (often useless) user manual. Searching for the user manual does not make it think you're looking for the service manual... it's absolutely idiotic, because who would search for "service manual" but actually want a user manual? Beyond some weird conspiracy theory involving anti-right-to-repair, I can't explain why.
I never, ever, for any reason, want to see results from the Daily Mail. I don’t even care if it’s the most relevant content: I just never want to see that site in my results. I know I can block it in hosts or whatever, but that’s not what I want. I don’t even want to see the link.
I wish DDG would give me this option.
And if you think the answer is to create and host a new, independent engine, you will be hard pressed to develop a good one, and to find the money to keep it running.
>Wikipedia is basically an information search engine without commerce or social media. The problem is that your results page is the article itself, and you need to scroll to the bottom to get to an external site.
>It's theoretically possible to change the UI or analyze Wikipedia to make a pretty solid search engine powered by millions of person-years of curation.
Using wikipedia as a data source for a "search engine" does not fulfill gp's (jacquesm) request of "One nice solution would be a search engine that throws out anything related to commerce or linking to commerce."
For example, go to the wikipedia page for the film "Groundhog Day" and look at the external links: https://en.wikipedia.org/wiki/Groundhog_Day#External_links
- one of the external links points to IMDB.com -- a commercial website owned by Amazon.
- another link points to Punxsatawney Groundhog Club website that's advertising a $30 beer tasting event (Hogtoberfest).
- another link points to a souvenir shop
To continue the example of commercial links, a lot of math articles on wikipedia point to MathWorld which has advertising for the commercial Wolfram Mathematica software package.
IMO, I don't believe a search engine based on wikipedia's list of external sites (even with ignoring the commercial links) is going to be useful for the mainstream audience. I probably do 50+ searches on google every single day and maybe 1% might be answerable from a "wikipedia search". For example, I needed a DIY answer to disassemble a Moen faucet and Wikipedia has zero articles with those instructions. In contrast, the Google (and Youtube) search results has the information I was looking for even though it has the unwanted ads and "content marketing" blogs from plumbing brands.
Rate limits.[0]
One can't really build a robust and comprehensive (server-side) search engine on top of Google's search API. In other words, Google Inc isn't going to provide an API so powerful that one can build an "alternative to Google" with it.
On the other hand, if you're talking about a "client-side" filter instead of a new search engine, (e.g. a client-side webscraper that flips through the Google results pages), you'd run into multiple problems. E.g. random CAPTCHAS screwing up the automation script, and the difficulty of creating a "rules engine" (e.g. machine learning) on your local computer that recognizes blog articles with a commercial slant.
[0] https://www.google.com/search?q=google+search+api+rate+limit
Google has created a disavow tool for this but for someone who runs a blog and not super tech savvy may not know about it and regardless it is still a burden for anyone to keep up with all this.
Again, I'm not an expert at this and I'm doubtful that a billion dollar search engine can be tricked by such shenanigans but it may be something worth looking into. if your site traffic drops
Edit: I'm often surprised how search engines fail to exclude obvious SEO farms. Especially looking at articles about stock, you frequently just find a ocean of what send to be clearly autogenerated articles.
Same tactic to find genuine content, but I find Google’s results to be better than Reddit’s own search.
Of course paid search introduces class issues. I think we should view search like a public utility because it’s absolutely necessary for surviving the world.
We missed out on like 90% of what the information revolution was supposed to bring to users, and instead the data giants have captured all the value.
Why do people prefer free over paid? I think people underestimate the influence of advertisers and underestimate the value of their own time spent researching and dealing with crappy products. It’s almost like a psychological bias.
I like the idea of a better user agent but would I pay for one? Judging by my refusal to pay for quality journalism I’m guessing it’d be a hard sell. That’s another thing I should be willing to pay for but don’t for some weird reason.
Plus the knowledge you can, and non-trivial technical expertise like: registering a domain, DNS configuration, setting up web software (even if "one click"), and maintaining it over time.
Do you have a link to this? I've tried googling "splicing a 500kW cable" (and variants) and haven't found much relevant.
I don't think this was an accident. Back in the 90's I paid for Netscape Navigator and for email. People used to even pay money to indexed by search engines (like paying to be in the Yellow Pages). Companies started "giving these things away" in a calculated move that I guess the nascent web population was just not cynical enough to reject.
Yeah but what does that even mean. How do you define commercial? Who is going to go through all of the pages and categorizing them into commercial and non-commercial? And how do you keep shills and “influencers” from littering your results with seemingly innocent but actually commercial content?
Keep in mind there is incentive to have a commercial meta tag so that when I’m actually looking for a good or service I’d be able to easily find it (or see the lack of it). The other 99% of the time ads and products are just an annoying distraction and a waste of money to serve.
TBH google kind of sucks for looking for stuff, amazon + reddit recommendations are my main source of surfacing things I want to buy.
Anything else seems to be served by the other providers just as well/poorly, and occasionally better.
Both were very precious and both are gone. There was no money in them I guess.
This needs to be fixed.
And not just the "shopping" tab but a fundamentally different set of algorithms for page ranking.
This sums it up nicely haha
Why shouldn’t the internet imitate actual life?
The frog can still sorta function. Wiggle in little circles. It ain't pretty or healthy or free. No hopping.
I wonder what it would be like to not have that nail through us.
No commerce -> no jobs -> unrest -> violence and wars (citation: centuries of human history).
Maybe, commerce is the nail which constrains humanity both in a good and a bad sense - by providing a peaceful alternative to violent lifestyles and by creating a consumerist culture?
When marketing convinces someone to buy something that ends up being a bad trade (i.e. the person that buys it is not made better off) then there is no such improvement and the world is not made better off.
Have you considered that maybe the majority of successful companies in the world are fulfilling customer wants and needs and thus making what you call Pareto improvements? And maybe it's just that the other "dumb people" derive joy from different stuff?
The market sorts these things out. It is mostly efficient. Not perfectly. But mostly. A company that is providing net-negative value to the world is a short-lived company.
Given that hucksters have existed for as long as markets have, I don't have your touching faith in the power of markets to eliminate them.
That’s not a citation. And given your thesis is a sprawling abstract claim, it really needs some real citations.
The likelihood for an individual to die by violence has been much greater for most of human history and prosperity and peace typically go hand in hand. I could provide references for these claims, but I think they are well known and references for the counter claim haven't been provided.
And evidence aside, your analysis is so coarse it’s basically impossible it could be true. Nothing that coarse is true.
And one last point: I have no doubt your head is filled with examples from history that back up your claim. What you almost certainly don’t have is evidence that there isn’t an equal raft of examples that contradict it.
Something like 'Save Money on Food', I am positive I am best-in-class, but I am only front page google, on occasion. Without a doubt, I should be the number 1 result, and instead, well SEO'd mom-science wins.
Btw, does this mean google should be looking for crappy code/design=genuine website?
Come to think of it, that might be kind of funny for a day or two.
I try searching for anything remotely bike related, bike community, mechanical information, or just general cool bike stuff and I can't find the human community underneath all the fluff articles trying to sell me shit. The internet, as free as it is, has been overflowed with commercial activity. Which pretty closely reflects the real world, but damn that's a shame.