It would likely be really niche, but it could become the anti-Google, which would be great when people actually seek an alternative to all the noise.
I think it has to be a non-profit like Wikipedia itself, I cannot imagine a model where it can also make money. The submitted site is a candidate but it has to improve the search quality as other commenters pointed out.
But based on the spam race, marketers will then tune content so that it doesn't trip those filters. Paid news and journal articles, etc.
But yes, even here on HN there are problems distinguishing between legitimate articles and paid news.
Also, I wonder if is it possible to detect promotional content by analyzing things like call to actions and such?
Of course what killed directories in the early web is that they had no hope of scaling.
You can have paid news, but it's not doing anything if the mark can't buy the product afterwards because you had to remove all associations with selling to get the news to rank.
Keeping marketing sites high in your SERPs would make you way more money on referrals though.
At the moment for web search it probably would not work, because I imagine from feature extraction to result there are several models involved to create intermediate results.
A search for "java twitter bot" places more emphasis on "bot" then on Java and then on twitter which is what a tf-idf would do.
A good start like you said but it's miles away even from yahoo or bing.
We are working on this scenario at http://www.shoten.xyz using document clustering and apache spark graphx+ giraph.
Ain't been an easy task so far but we have made some headway
BTW About 12 years ago I was building this search engine, and I was toying with the idea of building a classifier that classifies web pages based on their "genre" rather than category, so you can limit your search for shopping websites, forums, blogs, news sites, social media, etc. It was a bitch to train, and my classifier's algorithm was pretty crappy, but it showed some potential.
I think today modern search engine do that behind the scene, and try to diversify the result to include pages from multiple genres, but they usually don't let you choose.
Had some debate with myself if I should start by focusing on training for shopping pages (product pages & product reviews) - because that might make some money; or start by training for forums - which I'd enjoy a lot more. Or build a more general system which would definitely never work and never get finished.
Google actually let you filter by "discussions" until a few years ago, so they certainly do this kind of classification. It didn't work perfectly but sometimes did the trick. Don't know why they removed that feature.
Another perspective: people who find answers in forums are less likely to be interested in ads. And who knows, maybe making search shitty(in so many ways, not just formus), ad revenues rise ?
I would love something like this as well...
For most of my searches over the last year, Google has been so broken that it's almost unusable. At this point, to get any relevant results, I have to anticipate how Google will work, and then trial and error 10-15 times until I find what I'm looking for.
But, if I happen to be looking for incorrect information from 2005, Google works like a charm.
Same idea would probably work for online commenting. Vouch with a chain of responsibility. That's essentially how pagerank did its thing, but with no repercussions and vouching was automatic based on links from initial seed of what they thought was good. I'd do it with humans.
I would almost want the opposite. The best content on most topics I've found tends to be a page on a discussion forum of some sort, followed by blogs and more general editorial sites.
So on launch-day you would have checkboxes. On the one-year anniversary you´d have those checkboxes removed.
Google probably implemented this a couple of decades ago, though, so what we _really_ need is someone to come up with a new business model more attractive than Google's.
Ad-free is quite refreshing.
Whereby - a site would self-identify as being in a particular genre, say "healthcare" - and I could launch a tab to the engine and set my sub to "health, health-tech, healthcare, medicine, etc.." and then do my search and only those sites that set their category will show up in that search - but if I dont find my search, I can then easily slide out to other areas where I may not have thought what I was looking for would have identified with. Further - any post by any company/site could individually been given a topic to self-declare as... thus even if the company or site isnt necessarily in that space - their page or object could at least be a part of that result ranking....
Or has this been tried/found to be stupid?
You are describing the keywords meta tag.
While it is often told that competitors before Google did not use something like PageRank, which is not true, Google's PageRank algorithm was better and cheaper than the competitors' and effectively killed your idea 20 years ago.
But I find it slightly ironic that people are bitching about PageRank having slightly some issues with respect to the specificity of what they are searching for...
meaning that even though "killed this idea twenty years ago" we are coming back to the same problem...
Is that perhaps just due to the volume of info that is available on the web and the much more complex way we have categorized (mentally, not digitally) all the knowledge and information thats out there now?
What OP is saying that instead of pagerank we can have other ranking methods which is surely plausible.
We use tf-idf too but augment with page rank and clustering. gets more relevant docs
It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff.
Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related, for example. We've regressed.
My own half-solution is to look only for sites that are discussions - reddit, hn, etc. It could be better. A search engine that favored non-marketing content could really steal some thunder.
This doesn't look like that, but maybe its a start?