News for hackers. Stuff that matters.
Re:The Future of Science by nkoren on Sunday April 29, 2012 @01:38PM (#3906469)
On a related note, some years ago I read an academic paper -- alas, it was printed on a dead tree, and I can't find a link to a digital version of it -- which pointed out that the rate at which papers were cited was driven primarily by the rate at which papers were cited. This is not as much of a tautology as it sounds like.

Think about the process of writing a paper: you do some keyword searches for recent articles on related subject. You then look at the bibliographies for those articles, pick out whatever looks relavent to your topic, look at the bibliographies of those articles, etc. What this means is that apart from your initial keyword search, the primary criteria for including an article in your research is: "has it been cited already?". Relevance is merely a secondary filter.

This paper pointed out the effects of this phenomena: the vast majority of published scientific papers are never cited again; a moderate number are cited only a few times, and the remaining few -- having reached a bibliographic critical mass -- are cited thousands of times. The authors of the paper made a strong case that this was not a good reflection of the quality of research. In many cases, the process of reaching bibliographic critical was simply based on the almost random chance of acquiring those first few citations. The authors provided several examples of important scientific ideas which had been lost for decades, arguably because they had not attracted a critical mass of citations in the years immediately after publication.

In other words, humans suck at pagerank.

Anyhow, it occurred to me that this is a problem which could be solved with technology. Imagine an online word processor which -- in a sidebar -- suggests potentially related articles from ArXiv and Google Scholar. This would be based not on crawling bibliographies, but rather on semantic analysis of the adjacent paragraphs.

I think this would create some real benefits. It would remove much of the problems with citation bias, ensuring that important ideas aren't lost, and also that prior research isn't unwittingly duplicated. Wish I had time to implement something like this!

Re:The Future of Science by sb on Sunday April 29, 2012 @03:14PM (#3906740)
I think that your impression of doing related work is overly simplified. Since you are submitting to a conference/journal in your field, chances are high that the reviewers are knowledgable in the subject area and will point out errors in attributing due credits to related work.

While I agree that there are systemic problems with peer review and how the science "enterprise" works, there is fitting analogy from politics by Winston Churchill: "The worst form of government except for all the others."

Re:The Future of Science by nkoren on Sunday April 29, 2012 @05:16PM (#3907155)
I think you misunderstand. I'm not talking about attribution errors. I'm talking about the fact that discoverability and cross-linking is severely hampered the bias towards previously cited works. This doesn't create errors per se, but can narrow the scope of inquiry to the point where it becomes detrimental to the institution of science as a whole. It's a naturally emergent silo, but a silo nonetheless.

A NLP-based referencing system would not need to be correct all the time; it would merely need to be helpfully suggestive. As you're writing your paper, it would put tips in the sidebar: "Maybe this is relavent? (Hover to read abstract)". As long as there aren't an intolerable number of false positives, it would be quite a useful tool, I think.

Re:The Future of Science by sb on Sunday April 29, 2012 @08:08PM (#3907522)
Well, actually I was not only having attribution errors in mind, too. Peer review ensures that colleagues will tell you about related work that you don't know about. Sometimes, people will tell you that something is related, even though you yourself don't actually think it's related work. Only with some time and acceptance, you will see that the remarks are really related, probably not directly to your own contribution, but to the bigger field that you orient yourself in.

Come to think of it that this is probably the most important detractor for having an NLP based "recommender." Personally, something like this might be interesting, probably even a great help, but at the end of the day, people need to really read a lot of papers, follow the proceedings of their target conferences, journals, and asking colleagues for their bibliographies. This has the added benefit of teaching them how to present their own work in contrast to others, do meaningful evaluations (in the best of all worlds, of course!) and figure out who is doing interesting work and might be valuable to get into contact with. Of course, some parts could be automated, but there is currently no incentive for scientists to do so.

IMHO, it would be a much more important step for CS researches to publish their code, too, because I frequently come across papers that have no implementation or evaluation at all--and that's really bad, because then the least-publishable unit becomes an idea with nice pictures. Researchers can be very successful using this "publication strategy." Come to think of it, there should be another approach to rank scientists by the number of publications, or their impact; unfortunately, I have no idea what could work instead.

Re:The Future of Science by RichardPrice on Sunday April 29, 2012 @08:22PM (#3907553)
I totally agree about the importance of scientists to publish their code. That is critical. It's one of the many parts of the scientific process where the community would benefit from greater sharing.
Re:The Future of Science by eli_gottlieb on Sunday April 29, 2012 @11:07PM (#3907861)
Peer review ensures that colleagues will tell you about related work that you don't know about.

Not really, because other researchers are advancing their careers based on how often and how much they (a) publish and (b) get cited. So the colleagues most likely to be in a position to review your work are those who got cited a lot, who would primarily know about their work that got cited a lot.

The application of a relevance/NLP/PageRank-like additional-citation recommender program could come as a step in the reviewing process. Rather than having just human reviewers suggest further reading, a "machine reviewer" would as well, placing the query results in front of everyone involved in publishing a paper.

Re:The Future of Science by ktizo on Sunday April 29, 2012 @03:29PM (#3906789)
I think you may mean the worst form of government except all the others that have been tried.

We need to try more.

Re:The Future of Science by RichardPrice on Sunday April 29, 2012 @03:45PM (#3906846)
I think that is a great sentiment. To find systems that work a lot better than the current one, which is incredibly slow, we are going to need to try a lot of different ideas.

When you are venturing into the unknown, innovation is tough and challenging, but it's also very rewarding.

Re:The Future of Science by ajuc on Monday April 30, 2012 @06:05AM (#3908633)
Unfortunately trying new ideas when it comes to government, sometimes produces milions of victims.

But I agree - we should experiment more. One idea - we should be A/B testing any new law, before introducing it in the whole country.

Re:The Future of Science by ktizo on Monday April 30, 2012 @07:24AM (#3908754)
trying new ideas when it comes to government, sometimes produces milions of victims.

Usually only with the forms of government that don't care about millions of victims. Is really a no-brainer that any totalitarian system that intends to make an enemy of a large section of its population is probably going to be pretty awful. So I would strongly suggest weeding those forms out in advance, rather than using the common historical method of allowing them to be implemented through force by the paranoid and insane.

This is less of an issue when it comes to systems of improving feedback in science, unless it somehow enables an aspiring evil genius, or something.

Re:The Future of Science by gliese1337 on Sunday April 29, 2012 @02:50PM (#3906649)
I am always looking for ideas for software that I could work on that seems more useful than cat pictures, and this gets pretty high up in my estimation. Most of my minimal NLP expertise is being used up in developing software for foreign language instruction in the near future, but I'm definitely bookmarking this just in case I can ever take it up later.
Re:The Future of Science by sb on Sunday April 29, 2012 @03:20PM (#3906761)
I am not sure that this will work anytime soon, even if NLP gets better substantially. I think the problem reduces to examination of patents and their novelty. I think that having algorithms to find related work is going to be hard to the point of it being unreliable. For example, in CS there is a trend of changing nomenclature every n years, which I think makes it hard to find related work.
Re:The Future of Science by eli_gottlieb on Sunday April 29, 2012 @11:09PM (#3907864)
Google manages to serve advertisements despite changing nomenclature. We're just talking about "advertising" papers based on various data about the paper under submission.
Re:The Future of Science by nkoren on Sunday April 29, 2012 @05:18PM (#3907162)
It would be really great to see somebody take this on! Although if this idea makes you into a millionaire, then you'll owe me a burrito, okay?
Re:The Future of Science by lhnz on Sunday April 29, 2012 @04:07PM (#3906906)
Somebody needs to make a relevance algorithm which takes into consideration, (1) noise at low levels of citation activity, and (2) social proof being misleading at high levels of citation.