News for hackers. Stuff that matters.
Re:DeepDive – System that enables developers to analyze data... by nl on Thursday December 11, 2014 @05:29PM (#8737419)
It does probabilistic inference![1]

So many open source "Knowledge Graph"-y type projects concentrate on building them like databases, with a query language that assumes the data in them is correct. You see this in things like Freebase, DBPedia and Wikidata, where they typically end up in a triple store and you query using SPARQL.

This isn't how the real world works, and there isn't a lot of publicly available software that takes this into account. There aren't even than many papers about it (the Microsoft Probase paper is one, and there is work from Florida University(?) about using Markov chains to reason while taking probabilities into about).

I'm excited to take a look at this.

[1] http://deepdive.stanford.edu/doc/general/inference.html

Re:DeepDive – System that enables developers to analyze data... by anonetal on Thursday December 11, 2014 @09:09PM (#8738266)
Aside from the work on probabilistic inference, there is also many papers on "probabilistic databases" in the last 10 years (Chris did his PhD on that topic). That work has looked at SQL-style query processing over "uncertain"/"probabilistic" data.

These were some of the major projects: https://homes.cs.washington.edu/~suciu/project-mystiq.html, http://maybms.sourceforge.net/, http://infolab.stanford.edu/trio/, http://www.cs.umd.edu/~amol/PrDB/, http://dl.acm.org/citation.cfm?id=1376686.

Re:DeepDive – System that enables developers to analyze data... by nl on Thursday December 11, 2014 @09:21PM (#8738301)
This is a fair point. In a similar vein there is also BayesDB http://probcomp.csail.mit.edu/bayesdb/
Re:DeepDive – System that enables developers to analyze data... by peterlvilim on Friday December 12, 2014 @12:43AM (#8738740)
Re:DeepDive – System that enables developers to analyze data... by nl on Friday December 12, 2014 @02:24AM (#8738919)
Is it?

I thought BlinkDB was data warehousing on hdfs? I dont see any mention of inference-like features in the docs?

Re:DeepDive – System that enables developers to analyze data... by anonetal on Friday December 12, 2014 @09:55AM (#8740282)
It's not similar. BlinkDB builds upon the work on sampling-driven approximate SQL query processing (an early project in that space was AQUA@Bell Labs), and extends it to cloud/HDFS setting.

Although some terms come up in both places (e.g., confidence bounds, noise, etc), BlinkDB and probabilistic databases are fundamentally different from each other (I have worked on both topics).

Re:DeepDive – System that enables developers to analyze data... by tensor on Thursday December 11, 2014 @05:52PM (#8737516)
There is actually a huge body of work on probabilistic reasoning. Just do a google scholar search for the "probabilistic reasoning" or "probabilistic inference" or "probabilistic logic." You might have to dig around in the results, but there are definitely a lot of papers on the topic.

I hadn't heard about DeepDive until now, but I did previously come across another project that does probabilistic reasoning: http://alchemy.cs.washington.edu. I cannot speak to how they compare since I haven't looked into either in great depth yet.

Re:DeepDive – System that enables developers to analyze data... by nl on Thursday December 11, 2014 @06:00PM (#8737554)
Yes, I should clarify.

There is a lot of work on probabilistic inference. Huge volumes.

But then if you are interested on papers on using probabilistic inference on unstructured information there is a lot less information.

If you are looking for actual implementation reports there isn't much. The most interesting are the previously mentioned Probase paper, The Google Knowledge Graph paper and assorted IBM Watson stuff.

I've just realised that DeepDive is also the Wisci project[1], which is something I've been watching for a while

That Alchemy implementation requires a structured database of information, and doesn't really deal with how to get that.

[1] http://i.stanford.edu/hazy/home

Re:DeepDive – System that enables developers to analyze data... by pvnick on Friday December 12, 2014 @09:25AM (#8740111)
> there is work from Florida University(?) about using Markov chains to reason while taking probabilities into about.

You're talking about the work that Prof. Daisy Zhe Wang and her students are doing over at the DSR lab. Go gators!

[1] http://dsr.cise.ufl.edu/?page_id=250