It's that Google has destroyed their own search results in order to continue to expand their revenue opportunities.
If Google:
- Enabled downvoting on results, like YT videos. (Has its own spam problems, just like YT)
- Allowed you to block certain domains from your search results, like YT videos. (If they added some kind of "coordinated network detection" and down-ranked domains coordinating with ones you've blocked, that'd be pretty cool).
- Allowed you to create your own custom search engines, like "Programmable Search Engine".
That would be incredibly valuable. They already have most of the tech. They could even create a subscription service around custom search engines if they really wanted. Plenty of people would find something like that incredibly valuable.
Anyhow, buried in there is your startup idea. Remember: your startup doesn't have to generate the same revenue or profit as the incumbent on day one to be successful.
Internal Search owners can push for better algos, but what if the algo causes revenue to fall? Are there strong forces strong enough within the organisation to ensure that search quality prevails?
If this is the case, the problem is existential. It can only be arrested at the very top
This gets close to the real root of the issue -- attention is monetizable independently of the quality of content. There would be much less incentive to create SEO spam if search engines negatively weighted pages with ads and affiliate links, and if manufacturers were barred (e.g. by the FTC) from owning or imitating reviewers.
This is also something that Google can control if competitors come along.
i.e. If a reasonable competitor comes along that is willing to sacrifice ad revenue for better search result quality than Google, google can just adjust their search quality upwards to knock them out (and then adjust it back once the competitive threat is gone).
Perverse incentives from Google are all over the place - Searching for the delivery business "Just Eat" in the UK for instance returns an ad for their competitor Deliveroo above the legitimate organic search result for me - and I can also see that JustEat are trying to pay for their own brand name just to compete - and IMO this sort of behaviour is anti-competitive, borderline extortion considering Google is the de-facto way of searching for a business, and shocking from a search-quality perspective (where the wrong result is intentionally shown at the top because they paid more money).
Google's search algorithm is tuned up for searching the whole web. It turns out the heuristics you need are very different depending on the size of the collection.
When Gerard Salton was doing IR experiments with punched cards he was working with collections of as little as 70 documents and in that case you are going to be very concerned about recall and not precision. Maybe there is 1 relevant document and if you miss it you failed.
If you had 70 billion documents you might have 10,000 relevant documents and if you lost 60% of them you still have 4,000 documents. The end user gets more results than they can sift through.
Thus I always groan when I see a site is using "Google Site Search" because the relevance is usually worse than you'd get with the alternatives.
Connected with that is the tuning work: Google has sufficient data to tune up a big model for everybody but true personalized search eludes them because they don't have enough data from you to tune up a model for you.
Why would they do this? Google's customers are the advertisers, not the end-users. And no one is going to pay for a search engine, it's been tried and has failed.
Always curious about things like this. I certainly would pay for this; it sounds like many other people here as well would. I'm curious if the constraint is that there aren't enough people to actually pay for the investment required for the service, or if there aren't enough people willing to pay to meet the standard VC notions of success. We seem to have a problem with building and supplying services for niche (read: "not expressable as an integer percent of the world's population") customer bases, and I'm never sure if that's a business problem or a cultural problem.
Nobody is going to pay for /just/ a search engine. But they might pay for, say, a /better/ search engine, plus additional features around gmail/gcal/gdrive.
Think of it more as subscribing "to google" and less as subscribing to "google search".
Regardless, the point isn't to "fix" google. It's to highlight a possible path for a new market entrant.
... If an existing player wanted to make a move here, I would say that both Mozilla and Apple are well positioned to add "personalized search" to a subscription service. Same with Microsoft. DDG could also make moves here if they expanded beyond search.
A company with an objectively superior search engine could make even more money with ads so now you’re back to the beginning
An “objectively superior” search engine, from an end user’s perspective, might have to make engineering choices that come at the expense of ad revenue.
But we’re all just talking hypothesis, it’d be cool to see someone launch a startup to get some answers.
It's not open source as far I know and there's only the free trial way to try it.
Google could easily use its current fingerprinting to constrain (to an extent) multiple votes. Even knowing only a portion of the population will participate in the voting they can use a Wilson confidence interval[0] or similar to properly weight votes.
Random sampling works here since you're not guaranteed one vote per user per page and the outcome in binomial, seen and downvoted or seen and not downvoted.
[0] https://www.mikulskibartosz.name/wilson-score-in-python-exam...
They're going the OPPOSITE DIRECTION from this!! They recently removed all downvote visibility of YouTube videos from the user, so now downvotes only feed into their algorithm. So in the last line of defense of me ending up watching a shitty video, one of the most valuable tools has been removed by my betters. It's preposterous that people think that Google is doing a good job. They're actively getting worse, and ignoring everyone saying so.
All the big channels I follow are mad at this change, and there's a coordinated effort to bring it back.
you mean the dislike counter they just disabled to force people to sit through more low quality content and pre-roll ads to claim increase in platform engagement and viewership?
The only thing matters is revenue and Google had increases in acquisition costs in prior revenue reports. Expect to see the data points for the latter metrics to be highlighted on the earnings announcement, and a record quarter for YT coming out of the change.
I would love for Google to build this in. Until they do, there is a WebExtension that does this: https://addons.mozilla.org/en-US/firefox/addon/hohser/ ("Block or Highlight Search Engine Results"). I use it to block stuff like W3Schools so when I search for something, MDN is always #1. Saves me a lot of time having to add "MDN" to the end of every query.
Things have changed in the past few years, now that Google has developed advanced transformer models, but for a long time Google's question answering facility has been: "let spammers make 10^8 pages where the title is the question and the answer is in the page".
The trouble is that there's a fine line between "answer is in the page" and "word salad!"
Not convinced this would help. The spammers would just hire people to dislike competitors
> - Allowed you to block certain domains from your search results
This I would use. Never show me results form collider, watchmojo, ranker,
> - Allowed you to create your own custom search engines, like "Programmable Search Engine".
I think this would lead to people writing highly polarized engines. The Red Pill engine for example and we'd have a new problem, the proliferation of popular highly biased results. Of course that's not to say Google's results aren't already biased but they certainly are trying to cover everyone.
Are there any search engines that do this? It's a great, simple idea.
Couple of positions up or down in google results for somewhat popular and valuable keywords can mean the difference in thousands of dollars per day of ad or affiliate revenue. I suspect it would get pretty wild if google launched something like this. There already are black-hat seo methods and services, but something so simple and direct would turn it up to 11.
They already have the tech to fight this on YT. They, in theory, are supposed to be doing the same thing to detect inauthentic behavior on ad placement and click abuse.
Blocking pintrest would be a dream come true.
Not only that, Paul and Michael have seen plenty of startups, and at least in recent memory, the number of vertical search and consumer startups that Y Combinator has funded hasn't been that high
As a consumer startup, I know this issue firsthand. Paul and Michael assume that if you build a better product, they will come! That's simply not true these days.
Instead, you need to:
- Build a better product
- Option 1: Figure out a channel with enough growth on an existing platform. This likely means you're doing SEO for your new search engine
- Option 2: Get your customer lifetime value high enough so you can pay for ads. This is tough, since it's a bit of a chicken and the egg problem since most search engines are monetized with ads
As the founder of Wanderlog (YC W19; https://wanderlog.com), a consumer vacation planning app [1], I definitely remember the idealistic days when I thought the best consumer product on its own would win! But growth doesn't just come, and the same can be said of vertical-specific search engines.
[1] Try searching "[your city] itinerary" on Google vs. Bing: it's much more likely you'll find a small blog rather than Lonely Planet or the local travel bureau as the top result
The best part of it was (going to a foreign country) being able to find / identify all the attractions relative to each other, so I could go to cluster A on Monday, cluster B, on Tuesday, etc.
The hardest part of it (and why I needed to create a separate google sheets anyways) was--once I figured out opening hours of different locations, hard-to-book activities with limited reservations--the ease of moving things around more fluidly e.g. cluster B on Monday, cluster A on Tuesday, etc. and having a more information-dense view so I could see larger portions of the itinerary at once.
It would be cool to have an "input everything" --> "input time restrictions / unmovable things" --> output planned activity cluster type workflow.
Bing: https://i.judge.sh/ShareX/2022/01/www.bing.com_search_q%3Dat...
Google: https://i.judge.sh/ShareX/2022/01/www.google.com_search_q%3D...
Interestingly Google didn't have a top-result ad and the google.com/travel carousel is 4th from the bottom.
For the actual results, both thefearlessforeigner.com and paigemindsthegap.com seem to be actual travel blogs (the pictures didn't appear in a reverse image search, so they are probably organic), but they're clearly geared towards being a 'faq' for visiting the city and have affiliate links where appropriate. Bing went straight for discoveratlanta.com, and frommers.com is well-thought-out but not a personal travel blog.
nonoonononooonono. No. Don't monetize anything for the first 10 years. That's the only way it can work. Then you can go monetize it and buy an island and not give a shit if you destroy what you created.
Oh but don't worry. You'll have investors.
The hard part in all of this isn't finding and stopping spam - it's defining what spam is. Are all the pie recipes where there's a 2000 word essay about their grandma at the top 'spam'? They still have the recipe, and Google Home devices pick up the recipe instructions just fine so people end up not reading it, but many people would still consider that spam since it adds such an obstacle to getting the information you want. Same for cnet articles like "Best smart home devices to buy in 2022" - it's a reputable brand with a list of smart home devices, but it's hardly a review and exists to funnel people to their Amazon affiliate link.
This is one area where Google could use personalised results to provide a better experience for the user. Let me decide what spam is for me. Let me mark results as good or bad, so that the algorithm knows what kind of pages should be prioritised or filtered out the next time. Google SearchWiki was a step towards this but they killed it off.
We have seen what this leads to inside the social networks as well as YouTube, and at a macro scale I think we might want to have a shared concept of what constitutes a good search result for a given query.
At micro scale, it can seem more optimized to get exactly the type of result you want, but if we take an absurd example like an Apple Pie recipe shouldn't we all have shared understanding of what types of ingredients would make for an Apple Pie?
The shared understanding, I believe, is core to communication. If all of us have our own specific ideas of Apple Pie, then who is actually right on what an Apple Pie really is? What happens when your search results insist that an Apple Pie doesn't actually have apples in it, but instead pears?
Hmm why stop there let's actually make the users do the curating and even the content creation by rewarding them with social validation. Let’s have hard working moderators who work on the community full time.
Then we could just build a search engine over it. We could call it Reddit. Or HackerNews.
Maybe the users aren't all as good as professionals at curating the information. Let's hire professionally trained curators pay them well and we could call them newspapers. Then we can come in disrupt them and replace them with an algorithmic marketplace that eventually becomes infested with click bait.
(The fact that many of these books were rushed out with very poor quality control also didn't help.)
Google's algos, while advanced, still rely a ton on text to actually tell what the page is about. They need it.
If they just relied on other factors (title, links, website, etc.) they would end up with worse results for users. Im sure they've tested it.
Google's core algo in a lot of ways is much simpler than people think (in other ways of course it's very complex).
What's the point of writing succinct, to-the-point mini articles about problems and solutions if nobody finds them on Google?
I've done blogging for the last 10+ years, and many of those I spent as a freelancer working with startups/brands/editorials. Everyone is after "word count" and I absolutely hate it.
Whenever I work on articles for my own blog, I just don't consider word-count at all. I think if your content is great and informative, then readership will be natural.
Later on, I sold it because I needed the money. Not so much that I didn't want to keep working on it. Unfortunately, the new owners didn't have any idea how to maintain a "healthy" content blog, and it has plummeted down to around 30,000 monthly visitors. All the content they're publishing now is some thin headline-clickbait bullshit.
I even gave them free advice on how to fix it, but I think that for a lot of people, they just don't care and will mindlessly pump out as many pieces of content as possible. And such blogs can be identified from a mile away.
And therein lies the problem with Google SEO at the moment. Even myself, someone who has done SEO work for more than a decade, I can see that results are getting worse. In some niches, the same crappy articles that dominated 6-7 years ago are still dominant today.
I guess we're stuck in time, or so Google thinks.
Incidentally, prioritizing long content seems odd to me, in my experience the best pages are short and get right to the point, at least in the context of something like a recipe or other "how to" resources.
I think this isn't entirely related, but that's perhaps the beginning of a bias you might end up having that everyone experiences technology in the same way as it marches on. I've yet to encounter a Google Home in the wild, I imagine far more people are consuming recipes on phones, tablets and PCs.
At some point, Google must have moved away from using site-level reputation in search rankings, as I almost never see recipes from reputable sources like King Arthur Baking, Serious Eats, or Food52 in the first page of results.
It's like the bad-old-days of ExpertsExchange, which somehow was never delisted by Google for its shady SEO tactics.
As a user, your best personal and ethical move is to install an ad-blocker, to make ad-based business models less viable, which will help promote business models that don't abuse the customer.
This seems a bit overly cynical. Some search engines only served ads, but they're long gone. The survivors are those who dedicated themselves to finding links which were responsive to people's search intent. They seem to have gotten into ads because it was the best business model in this market.
This is really what made me suspect that Google was teetering on the edge of the MBA death spiral: these problems run for years when they'd be easy to block, which suggests to me that whatever metric gets you a bonus / promoted doesn't include things like that which are long-term threats to their core business even if it's selling a lot of ads short-term.
Some even made slideshows of SO screen captures and put that on Youtube, with a fake video or spoken intro to make believe an actual content will be discussed... A number of shameless people would go any length to grab bits of money anywhere and anyhow, and I've hit those links a couple of times.
To put it simply, a new generation of the people who used to make the reliable niche websites that not just answered your questions but also helped you learn a particular topic have moved to youtube instead.
Google search is hollowing out as a result with the meat going and the SEO'd fluff that kinda answers the question but ONLY the direct question being asked with none of the wider expertise that more educated people in what they were searching for.
Of course google owns youtube as well.. so perhaps they just see it as an inevitable transition.
(Disclosure: I work at Google, but not on search)
I still think the model could work if the algorithm is sufficiently different than Google's. Ideally, people would go "I did not find anything I cared about on Google, I know, I'll use Bing!" - but nobody does this, because the results are consistently worse.
Don't get me wrong, I like G as a company, I think they do worthwhile things! But they have left things slip and need competition into this field, I mean real competition, then maybe they would actually address issues.
Maybe the issue is also on the incentive level as well. I mean more searches means more eyeballs and more money for Google. If someone searches one thing and they are done that is less interaction! I hope they don't work like this, but it's possible.
And another possible problem is the opposite. Maybe Google is optimizing search for what it thinks people want, but it uses the wrong metric. Or it gives people what they want but not what they need.
There is no scenario - none - where thousands of engineers at Google working on search wake up in the morning and say "we sure have made it good enough wr2 SPAM. I think I'll have another Danish."
I don't claim that Google has any of these and certainly have no insight into their search group. But I've personally been at powerful companies with best-of-the-best talent that were blind to the decay in their own living room, so I would caution against immediate dismissal of PG's take.
I think it’s more likely that because they are just building hundreds of tiny tweak experiments and it’s someone else who desides what to build and if it even worked. Search quality is such a meta-problem that it goes beyond any real hope of simply working on it in anything beyond piecemeal trial and error fashion on their dataset.
The complaints I've read are from exactly the kind of generated content farms people are complaining about in this thread.
The fact that this feature does not exist shows that there is something deep within Google's core that is preventing them from addressing SEO spam, just like there is something deep within Airbnb that makes it difficult to filter out Airbnbs with problem reviews.
Google has been coasting for a good long time and now major players are realizing they are wide open for disruption.
The reality is more that some Google engineer will come up with an algorithm change that makes the result 40% better, but it will come at the expense of making that search 3ms slower so the change won't get merged. Or it will make the results worse for some niche set of queries that the business team really cares about, so again it won't get merged.
There are lots of consumers who would gladly pay $1 a month or whatever in order to use a couple extra milliseconds of compute power per per search in exchange for drastically better results, so there is lots of room for a startup to compete.
Google has a paid-for Search API, so they could do that if they chose to pursue it. And then they could let Google One users opt-in to the same thing via ordinary Search. I'm not sure whether Bing has anything equivalent.
I think a really good search engine would still algorithmically search it's index, but the content library should be human-curated with a goal of ingesting content via author, not via platform. Once a given author was human-approved as a quality source of information, content they produce could be automatically ingested going forwards, and conditionally re-reviewed by a human if there were reports the quality had decreased.
I think the death of the directory search dramatically dropped the number of self-curated, informative sites from a domain expert that were common in the early internet. Now instead of making a website, many people are on content silos like Reddit/FB
Assuming a method also existed for an author to authenticate themselves with the search engine, one could also enable an author to help identify their content across multiple platforms, as well as suggest other quality authors to consider.
That may be true, but I think one of the good points made on the OP is that it might actually be cultural constraints that keep them from solving the problem:
https://twitter.com/paulg/status/1477761335412809729:
> You might need to do a lot of manual spam fighting initially. That could be both the thing-that-doesn't-scale, and the thing that differentiates you by being alien to Google's DNA. (They must hate manual interventions; so inelegant).
Google has some very smart and knowledgeable people, but the things they do have to fit into certain boxes, which means there are some problems they just can't fix, e.g.
* Everything has to be automated at scale, which leads to consistent poor user experience (unappealable account closures initiated by inscrutable algorithms, SEO spam).
* You get promoted by building new products, not maintaining existing ones, which leads to self-defeating churn outside of core areas (e.g. abandoning Google Talk and squandering their position in the messenger market).
* etc.
Of course you can always hardcode the site you want in the Google search results but this is hacky and not very expressive.
These sort of things are easy to spot, but only if you actually have a basic amount of familiarly with the topic. It's hard to spot with "AI" or super-cheap labor.
Google is smart so I assume they crunched the numbers and figured out they make more money from people filtering through crappy results that include viewing and clicking ads than by surfacing good content.
I think Google is optimizing for ad revenue, not for good search.
Really?
I can point you to Hard Problems that have been solved better at little startups than at Google - or, indeed, at any other bigco. That's why acquisitions happen.
Why does Google having 1000 engineers working on a problem automatically mean they are the smartest?
This is def a user-engagement strategy -- but it has cons as well.
Part of the complaints in the thread were spam related, other were something deeper
The only advantage a startup might have is that they could do completely new concepts, such as specifying what area you search in, allow you to modify their classification of your query and/or moderating sites you include - which is probably necessary anyway, since you'll hardly have the budget to fully index the web. I'm not saying it's impossible, but it's not going to be easy at all.
And after all of that, you still need a way to make some money.
It takes time and effort to build up a spam site's ranking but it is trivial to blacklist those who get to the top.
This is hyperbolic, right? But they can solve spam in a split second, if they just admit they're watching you all the time.
[edit] /s thx for reading to the end, folks.
I'm not saying that curated search results for particular verticals is a terrible idea (though I'm sure like anything the devil is in the details), but on the whole Google search is very, very good considering the constant assault they are under from spammers (which most other search engines are not, at least directly).