News for hackers. Stuff that matters.
Re:Wikipedia starts work on $2.5M internet search engine pro... by contingencies on Sunday February 14, 2016 @11:02PM (#11101444)
1) Public curation mechanisms for quality;

The Mozilla / open directory project tried this. Curation doesn't scale and often assumes a single unifying ontology. This is particularly problematic in a cross-cultural context. Besides, 'quality' is not a unidimensional metric in a result set: consider timeliness, authority, notability, uniqueness, comprehensibility, etc.

2) Transparency, telling users exactly how the information originated;

Most search engines include a URL, I can see a [crawldate] button like the [cache] or [translate] buttons on each hit adding some information, but this will be of dubious additional utility for most searches.

3) Open data access to metadata, giving users the exact date source of the information;

As above.

4) Protected user privacy, with their searching protected by strict privacy controls;

We have duckduckgo already, friends are welcome but it's hardly a unique offering nor a trustworthy one given Snowden's revelations regarding the scale of systematic 5 eyes traffic monitoring/recording.

5) No advertising, which assures the free flow of information and a complete separation from commercial interests;

DDG or Google or Bing with plugins can supply this. Not ground breaking.

6) Internalization, which emphasizes community building and the sharing of information instead of a top-down approach.

This is so amorphous as to be a non-point.

So out of six points, 2 things (33%) are only useful in edge cases, 1 thing (16%) is too vague to be useful, and the other 3 things (50%) are currently implemented by others and have been tried before.

I would like to see the input of the former Blekko guys on this, https://news.ycombinator.com/user?id=ChuckMcM + https://news.ycombinator.com/user?id=greglindahl

Re:Wikipedia starts work on $2.5M internet search engine pro... by abalone on Monday February 15, 2016 @12:17AM (#11101644)
> Curation doesn't scale and often assumes a single unifying ontology

Wikipedia is a pretty big exception to that assertion. Perhaps DMOZ (a clone of Yahoo circa 1996) is not the only way to do curation. Perhaps Wikipedia could apply what has worked for Wikipedia, i.e. develop a set of POV-neutral criteria for organizing collections of links and then invite everyone to participate.

It's really easy to be negative. But that's something that might at least be an interesting research project for the #1 open-curation system in the world.

Re:Wikipedia starts work on $2.5M internet search engine pro... by contingencies on Monday February 15, 2016 @12:39AM (#11101695)
You make a fair point. I'm not rubbishing Wikipedia, just questioning the supposed USP. I would also point out in response to your argument that a Wikipedia article and a set of search results are apples and oranges.

The article is written once then modified or evolved occasionally by (almost exclusively) humans, but very frequently read. It is intended to be intelligible, being structured and based in natural language. It has a very well defined scope within a flat namespace, and often clear relations to multiple formal ontologies. It is structured to be consumed in part or in whole, and may contain rich media and strong supporting contextual information (related pages).

By contrast a search result summarizes a set of potential information sources that may answer a search query in whole or in part, to various definitions of "answer". It is generally written once, by a computer, and thrown away after some period of caching. It is intended to be concise. Each component result has relatively poor context, relying upon the searcher to interpret timeliness, authority, notability, uniqueness, comprehensibility, etc. with the limited information presented, typically a very short content excerpt. It is structured to be scanned, classically in a ranked fashion from "best hit" to "worst hit", and is generally a wall of text.

Wikipedia successfully attracts people to contribute to the former, but the latter - where the information product is generated on the fly and lasting impact is amorphous (nothing particularly concrete for contributors to point to and say "I did that! Warm and fuzzies!") - is a very different beast.

I too believe there is room for innovation ... there are potentially low hanging fruit like inter-linguistic semantic queries (not keyword search) ... but there are no such key problem areas identified in the paper's summary.

Re:Wikipedia starts work on $2.5M internet search engine pro... by notahacker on Monday February 15, 2016 @07:46AM (#11103031)
The other big problem is that curating search results is inherently about prioritising a position rather than establishing a sourced and reasonably neutral version of the truth.

I'm imagining the edit wars and debates that take place on contentious wordings or facts in some parts of Wikipedia, but on a much wider scale involving hundreds of SEO consultants each aware that changing a particular criterion will have a quantifiable impact on their clients' bottom line. It doesn't sound like it would be fun to police.

Re:Wikipedia starts work on $2.5M internet search engine pro... by abalone on Monday February 15, 2016 @05:14PM (#11106452)
Wikipedia already curates links to some extent on every page under "External Links". So there is a seed there.

And even the page text is not immune from the problem you describe. Grading and prioritizing sources is a fundamental part of producing a "reasonably neutral version of the truth." It's what determines what gets cited and how prominently it influences the article.

So while I wouldn't equate text and links in terms of the difficulty of managing POV-neutrality, I would say they sit on a spectrum.

Re:Wikipedia starts work on $2.5M internet search engine pro... by Animats on Monday February 15, 2016 @03:59PM (#11105920)
There was a remark recently that most Jeopardy answers are Wikipedia titles. Consider Wikipedia as an ontology, with Wikipedia titles as the vocabulary. A search engine could associate articles with relevant Wikipedia titles, and try to do the same with queries. The first step of search is then relatively straightforward.
Re:Wikipedia starts work on $2.5M internet search engine pro... by frik on Monday February 15, 2016 @12:42AM (#11101701)
> We have duckduckgo already

DuckDuckGo a meta-search-engine! It relies mainly on Yahoo Boss API which uses Bing search (for most countries)! Yahoo Boss API turned from free to expensive in early 2015 and the future of Yahoo (tech company, not Alibaba stock) is uncertain.

We definitely need more search engines, only 6-7 exist that cover a wider range (international). Search on HN to retrieve the list, we had this discussion before.

Re:Wikipedia starts work on $2.5M internet search engine pro... by contingencies on Monday February 15, 2016 @12:49AM (#11101719)
I agree with you, but this quote was in the context of the claim of a privacy USP. You have taken it out of context.
Re:Wikipedia starts work on $2.5M internet search engine pro... by selectodude on Monday February 15, 2016 @01:28AM (#11101816)
BOSS is being discontinued on March 31st, so I imagine that DDG has some sort of idea what they're going to be doing after that date.

Any insight?

Re:Wikipedia starts work on $2.5M internet search engine pro... by infbtag on Monday February 15, 2016 @04:57AM (#11102476)
They could use the Bing API directly or just stick with Yandex.

http://datamarket.azure.com/dataset/bing/search

Re:Wikipedia starts work on $2.5M internet search engine pro... by xojoc on Monday February 15, 2016 @01:53AM (#11101879)
They are using Yandex now with a bit of their own crawling.
Re:Wikipedia starts work on $2.5M internet search engine pro... by greglindahl on Monday February 15, 2016 @03:50PM (#11105846)
Curation scales for some topics. It was hard to build a curated list of Linux sites, they come and go. But there are only a couple of new good, comprehensive health websites per year.

I was Blekko's founder/cto. And it's worth noting that our founding team was the Open Directory Project's founding team. Blekko's curation data was even better than dmoz in its day. Check it out: https://github.com/blekko/slashtag-data

Re:Wikipedia starts work on $2.5M internet search engine pro... by bobajeff on Sunday February 14, 2016 @11:17PM (#11101483)
Duckduckgo is ad free? I never knew this. How do they make money?
Re:Wikipedia starts work on $2.5M internet search engine pro... by contingencies on Sunday February 14, 2016 @11:19PM (#11101492)
My mistake. You are right, they do have ads, I just always use a blocker. I've updated the text.
Re:Wikipedia starts work on $2.5M internet search engine pro... by stephenr on Monday February 15, 2016 @12:12AM (#11101632)
Ddg has an option to disable ads, they just ask that you help promote them.