News for hackers. Stuff that matters.
Re:Wikipedia Is Nearing Completion, in a Sense by britta on Sunday October 28, 2012 @08:24PM (#4710467)
The notability guideline (http://en.wikipedia.org/wiki/Wikipedia:Notability) is closely related to the verifiability policy (http://en.wikipedia.org/wiki/Wikipedia:Verifiability) — from Wikipedia's perspective, if a subject has not been covered by multiple reasonably reliable secondary sources (in other words, if it isn't "notable"), we can't write a reasonably verifiable article about that subject. Every article has to include secondary sources as references, so that editors and readers can quickly fact-check.

There are plenty of thoughtful discussions elsewhere about why these notability and verifiability rules are flawed, but I wanted to point out this connection because it shows that the notability guideline is possibly not as arbitrary as it seems at first. I wish I could figure out a way to help Wikipedia explain this better. Maybe "notability" is the wrong name for this guideline.

Re:Wikipedia Is Nearing Completion, in a Sense by mjn on Monday October 29, 2012 @12:15AM (#4711092)
I agree with you as it currently stands, though I think there's been a shift on that. There used to be an attempt to enforce an independent notability guidelines, that Wikipedia would only cover subjects that were "encyclopedic". I believe that even predates the verifiability/referencing push. But where I agree is that I believe the latter has mostly obsoleted the former: nowadays, if you have good references for your article, it's ipso facto notable.

I wrote a bit about that evolution elsewhere, on the off chance it's interesting: http://www.kmjn.org/notes/wikipedia_notability_verifiability...

Re:Wikipedia Is Nearing Completion, in a Sense by saurik on Monday October 29, 2012 @04:33AM (#4711498)
> ...if a subject has not been covered by multiple reasonably reliable secondary sources (in other words, if it isn't "notable"), we can't write a reasonably verifiable article about that subject...

It is nearly impossible to verify an article based on secondary sources. To take a non-Wikipedia view of this for a second, I went and pulled the descriptions of these sources from academic institutions. Princeton's reference desk describes a secondary source as follows. After that, a similar description from the UCSC library.

> A secondary source interprets and analyzes primary sources. These sources are one or more steps removed from the event. Secondary sources may have pictures, quotes or graphics of primary sources in them.

> The function of these is to interpret primary sources, and so can be described as at least one step removed from the event or phenomenon under review. Secondary source materials, then, interpret, assign value to, conjecture upon, and draw conclusions about the events reported in primary sources.

(At this point I also feel it important to point out that both of these sources believe that encyclopedias are "secondary sources". If you do a search on Google for "tertiary source", which Wikipedia adamantly believes is a third category that includes encyclopedias, you get only 43k hits, half of which mention "wikipedia". The few universities that mention "tertiary source" list encyclopedias as being in both categories: only Wikipedia seems to believe that encyclopedias are clearly and definitively a "tertiary source".)

If you are interested in opinions and analysis, you can happily refer to a secondary source, but if you want to verify what actually happened, you cannot be taken seriously unless you can show a clear trail of evidence that terminates in a primary source.

(Certainly, if it is impossible to obtain primary sources, then one can use a secondary source, but that is something that should only be used as a last resort: if you have access to the primary sources that a secondary source used, you should verify them yourself.)

The result of attempting to find truth from secondary sources is that you will forever be plagued by horrible bias, both in terms of what things you can find to be "notable" and in terms of the validity of the opinions you can find in them.

Worse, as encyclopedias--which today pretty much means "Wikipedia" to a very large percentage of the public--are used as reference material for the construction of secondary sources, everything from newspaper articles to books, claiming secondary sources to have anything to do with "validation" just makes your chain of evidence circular.

And, in fact, there have been multiple published large-scale examples of such circular information ending up in Wikipedia, as temporarily un-sourced information gets used in newspaper articles which then reinforces the information in Wikipedia when editors attempt to find sources.

Regardless, the entire notion is kind of preposterous anyway, as the way that most secondary sources operate they just print what they are told by the people they interview for articles without the citations required to verify the ultimate source of the information.

This means that if someone, whether it be the Roth example from a few days ago or anyone else wanting to provide a paper-trail for information on Wikipedia, wants to be able to get something into Wikipedia, they really just need to be in the position to tell a reporter the information: they do not need to post it themselves directly.

As an interesting contextual example of this, Wikipedia now actually does have an article on Diggnation. In this article, it states quite firmly that "there are an estimated 250,000 regular subscribers to the show" along with a sourced citation to... the New York Times.

However, the article in the New York Times quite clearly was entirely itself sourced by simply asking the people who worked for Diggnation and Revision3 a bunch of questions; it even quite clearly states that that number came from them: "Revision3 says it counts roughly 250,000 views each week".

In such a situation, the way the New York Times operates (and I know this first hand, as there have been articles about the things that I do published by them) is that they will happily publish whatever number you tell them, and at best "fact check" it by calling you back to verify they heard you correctly.

This does not make the number true; it barely even demonstrates "Diggnation was important enough to have an article written about them", due to the "slows news day" phenomenon. The idea that this source is somehow different from any other source is just a fantasy, and one that, as far as I've been able to tell, one that only Wikipedia believes.

Wikipedia, in fact, seems to take the exact opposite stance on all of this, claiming that "Wikipedia articles should be based on reliable, published secondary sources and, to a lesser extent, on tertiary sources", along with a long list of rules about how primary sources (which are apparently even more dangerous in their world-view than tertiary sources) can technically be used, but only in highly limited and nearly useless circumstances.

http://en.wikipedia.org/wiki/Wikipedia:No_original_research

The result of this insanity is then numerous situations like the one that treelovinhippie ran into. Re-telling him what the rules of Wikipedia are--when it is quite clear that he got to experience them first-hand and was sufficiently unimpressed as to ask for their removal--seems to be missing the point: for this purpose, the rule really is "arbitrary", and couldn't possibly have anything to do with "verifiability"; if anything, Wikipedia's rules on "notability" at the same time cause numerous topics to be unable to be covered and cause the information that does get published to be based on shaky foundations and unreliable sources.

Re:Wikipedia Is Nearing Completion, in a Sense by ErrantX on Monday October 29, 2012 @06:32AM (#4711696)
Worse, as encyclopedias--which today pretty much means "Wikipedia" to a very large percentage of the public--are used as reference material for the construction of secondary sources, everything from newspaper articles to books, claiming secondary sources to have anything to do with "validation" just makes your chain of evidence circular.

This is a phenomen Wikipedians are aware of. But what you refer to here are often tertiary sources, and not necessarily the sort of material you'd want to use.

The result of attempting to find truth from secondary sources is that you will forever be plagued by horrible bias, both in terms of what things you can find to be "notable" and in terms of the validity of the opinions you can find in them.

You're arguing to swap that bias for your own? i.e. you look at primary source material and decide which items are notable for their own article. What about if another editor disagrees?

In academia secondary sources are an established way of processing primary information; an expert reviews the primary material and, citing it, draws conclusions. Other experts may do the same, and disagree with the first one.

Wikipedia is a tertiary source, which draws on secondary and, yes, primary material to summarise the current status of a topic.

However, the article in the New York Times quite clearly was entirely itself sourced by simply asking the people who worked for Diggnation and Revision3 a bunch of questions; it even quite clearly states that that number came from them: "Revision3 says it counts roughly 250,000 views each week".

In such a situation, the way the New York Times operates (and I know this first hand, as there have been articles about the things that I do published by them) is that they will happily publish whatever number you tell them, and at best "fact check" it by calling you back to verify they heard you correctly.

This is a problem, and one with no easy solution. The approach of requiring sources to have their own review process (i.e. media has editors who theoretically vet content) should fix this, but as you point out, in practice it is not 100% reliable.

Re:Wikipedia Is Nearing Completion, in a Sense by saurik on Monday October 29, 2012 @07:14AM (#4711775)
> You're arguing to swap that bias for your own? i.e. you look at primary source material and decide which items are notable for their own article. What about if another editor disagrees?

treelovinhippie's stated opinion was "remove the "must be noteworthy" rule" (from which I am mentally rewriting his statement to "notability"). I personally agree with that, as I do not believe that the notability requirement is leading to increased article trustworthiness.

I would rather see the idea of "notability" replaced with an "article score" based upon the history: there have been a few glorious visualizers for Wikipedia designed to really make this information hit center, and I think any of them would be a better global solution than "notability".

> In academia secondary sources are an established way of processing primary information; an expert reviews the primary material and, citing it, draws conclusions. Other experts may do the same, and disagree with the first one.

I imagine this is determined heavily by the field? I can't imagine in either mathematics or algorithmic computer science there being any need to wait until a review paper is published "drawing conclusions" (whatever that would mean) from the information in the primary research.

However, I honestly feel like you are dragging me down an academic rathole in these very specific cases, such as medicine: the kind of material that we are explicitly discussing here, Diggnation, is not going to be covered in anything remotely approaching a scholarly source.

Instead, what we are comparing are situations like 1) referencing a New York Times article stating that "person Y said X" and 2) directly linking to the blog or Twitter feed of person Y and demonstrating first-hand that they said "X": in these situations, removing a layer of required trust.

In my experience discussing situations like this with Wikipedia editors, in addition to my direct attempts to read through the rules that Wikipedia publishes for how their website should be used, it is quite clear that Wikipedia would prefer you to cite the NYT article rather than hotlink the information.

> This is a problem, and one with no easy solution. The approach of requiring sources to have their own review process (i.e. media has editors who theoretically vet content) should fix this, but as you point out, in practice it is not 100% reliable.

So, Wikipedia specifically states that "mainstream newspapers" are an example of "the most reliable sources". Can you provide me a reference to where Wikipedia is claiming that you can only depend on sources that "have their own review process"? (Or, was that just a potential idea for a fix?)

Personally, in this situation, I would much prefer a reality where Wikipedia simply admits that secondary sources (in this field; again: not medicine, and possibly not even history) are nothing more than opinions (in the general case, this is how Wikipedia defines secondary sources: opinions and interpretation).

If they did this, then they couldn't cite a New York Times article for a sentence that simply stated a fact: they could only cite a New York Times article and state "the New York Times states", for which the New York Times should be a perfectly valid (and I will argue "authoritative") reference.

Yes: in this case that would probably make it impossible to just state "Diggnation has 250,000 subscribers" without some kind of in-article qualification; however, I don't think that that is inherently a bad thing: I feel like if I were writing a scholarly article on this subject, I wouldn't be able to say anything definitive either.

Re:Wikipedia Is Nearing Completion, in a Sense by ErrantX on Monday October 29, 2012 @07:34AM (#4711810)
I personally agree with that, as I do not believe that the notability requirement is leading to increased article trustworthiness.

I struggle to follow this because Wikipedia doesn't consider "notability" to relate to article trustworthiness. It's instead intended to act as a soft margin for the topics that deserve a standalone article (remember, noteworthiness is only related to whether an article should exist or not).

I would rather see the idea of "notability" replaced with an "article score" based upon the history

Interesting idea, and I'd like to hear more of this idea! However, from this I suspect you are mixing up the reliability of an article and the specific phrase "notability" (about how deserving a topic is to have an article). When deciding if an article is notable it will usually have little or no history!

I imagine this is determined heavily by the field?

Yes, a point Wikipedia admits.

I can't imagine in either mathematics or algorithmic computer science there being any need to wait until a review paper is published "drawing conclusions" (whatever that would mean) from the information in the primary research.

Then you would, I am afraid, be wrong :) In such a field many papers are published contending a theory, and then other experts review the contention and submit responses/reviews/criticism.

the kind of material that we are explicitly discussing here, Diggnation, is not going to be covered in anything remotely approaching a scholarly source.

Yes, which is the key to the problem.

Can you provide me a reference to where Wikipedia is claiming that you can only depend on sources that "have their own review process"?

Yes, the core sourcing policy requires a "reliable source" (http://en.wikipedia.org/wiki/Wikipedia:RS). "Articles should be based on reliable, third-party, published sources with a reputation for fact-checking and accuracy". http://en.wikipedia.org/wiki/Wikipedia:SOURCES#Reliable_sour... goes into more detail.

Instead, what we are comparing are situations like 1) referencing a New York Times article stating that "person Y said X" and 2) directly linking to the blog or Twitter feed of person Y and demonstrating first-hand that they said "X": in these situations, removing a layer of required trust.

Is it? Now I agree that if the NYT has simply asked Diggnation for those figures then Wikipedia should note that is where it comes from. However citing it to NYT is intended to demonstrate that someone other than the editor inserting the material trusts its veracity. Its difficult because we simply do not know where NYT got that info: if Diggnation showed them some real figures (say, a screenshot) then the NYT article is better than a tweet!

Which is why this is not a simple problem.

Re:Wikipedia Is Nearing Completion, in a Sense by saurik on Monday October 29, 2012 @08:19AM (#4711905)
> I personally agree with that, as I do not believe that the notability requirement is leading to increased article trustworthiness.

> Interesting idea, and I'd like to hear more of this idea! However, from this I suspect you are mixing up the reliability of an article and the specific phrase "notability" (about how deserving a topic is to have an article). When deciding if an article is notable it will usually have little or no history!

The reason I am "mixing this up" is that the only argument I have ever heard for why Wikipedia needs a notability policy at all is the one that was cited by britta earlier in this thread: that without such a clause there would be tons of articles that are hardly ever looked at by anyone, hardly ever edited by anyone, and containing information that is difficult to verify. I believe that there are numerous better solutions to this than attempting to use the "notability" filter, as I maintain that "notability" does not actually lead to "veracity".

To be very clear about this, I will repeat the context from britta that started my involvement in this discussion:

>> The notability guideline (http://en.wikipedia.org/wiki/Wikipedia:Notability) is closely related to the verifiability policy (http://en.wikipedia.org/wiki/Wikipedia:Verifiability) — from Wikipedia's perspective, if a subject has not been covered by multiple reasonably reliable secondary sources (in other words, if it isn't "notable"), we can't write a reasonably verifiable article about that subject. Every article has to include secondary sources as references, so that editors and readers can quickly fact-check.

Going back to your response:

> Its difficult because we simply do not know where NYT got that info: if Diggnation showed them some real figures (say, a screenshot) then the NYT article is better than a tweet!

I must apologize here, as I was intending to be using a different example, but you took me to mean the Diggnation example: this is my fault, I should have been more clear.

What was coming to mind with relation to the "Twitter post" example, is that there is a ton of journalism on topics I directly care about that is based on the Twitter feeds of people I work with: a lot of "person X said Y", which is then translated into some article on what is or is not possible with a tool that person X builds. The fact that a reporter read that statement and repeated it doesn't make it more true, and in fact the opposite is quite common: they are paying sufficiently little attention and have sufficiently little background knowledge that they repeat it wrong.

Please understand: I am not talking about a situation of "interpretation" or "research", but more like establishing dates on when things happened... I understand that this feels pretty blurry (but it seems equally silly to go into a detailed example of something where my example is itself biased; I am happier sticking with the examples such as Diggnation and RSA); it is simply a situation where "that dude at Wired that finds this stuff gets him readers" is not somehow more trustworthy than the place he got the information from... either the information shouldn't be published at all (I believe this is probably a quite reasonable course of action), or "that dude at Wired" should be skipped and the original source should be used.

That said, part of your comment doesn't rely on that misunderstanding I caused: it is true that we don't know where the NYT got that information, however that uncertainty really doesn't make it any more true; while it means there is some possibility that the information was gathered in a way that we should indirectly trust, as we can't see it we can't verify it, and it is honestly not in any way better than if some random person on Wikipedia just asserted it to us... reporters for major publications (such as the New York Times and Washington Post, both of which I have first-party experience with due to articles written about my work) really do trust that you are a reliable source on things that you control, and really do attempt to fact check by calling you back for verification.

Re:Wikipedia Is Nearing Completion, in a Sense by ErrantX on Monday October 29, 2012 @09:40AM (#4712173)
Ok, fair enough. Britta touches on some of the issues, but not in its entirety. Notability is about a) requiring there be at least one reliable third party source (so that the article has a chance of containing verifiable information) and b) ensuring that there is some limit on the scope of Wikipedia. It is this latter one that is the key facet.

Whilst Notability is closely related to Verifiability it is not quite in the way britta cast it, but rather related to requiring the material used to define a subject notable (i.e. a significant claim to importance) is verifiable. i.e. the relationship works the opposite way.

I was intending to be using a different example, but you took me to mean the Diggnation example: this is my fault, I should have been more clear

Ah, my apologies, I'm reading quickly as it is a busy day.

Please don't get me wrong; the issue you highlight is a major problem, one I have raised a few times internally with the community. But there has been no easy resolution.

It's worth noting that the reliable sources policy explicitly notes that reliability hinges on not only the publisher but also the content and the author. If an author is seen to lack the qualifications, or has a bad reputation, these factor into consideration.

With that said; a lot of Wikipedians don't know this. A problem I run into constantly when discussing sources ("Well, it was published by the NYT's so it doesn't matter what their reputation is"). It's not the policy at fault there, but the lack of interest of our own community in the rules...

"that dude at Wired that finds this stuff gets him readers" is not somehow more trustworthy than the place he got the information from...

The intent of the policies (and bear in mind what I say above as to how much that holds up..) is that the secondary source is used to filter what in the primary material is considered important to the wider community. To take an example: when Microsoft released Windows 8 there was quite an extensive list of new features. Simply recording that isn't what Wikipedia aims to do, instead you would use secondary sources to highlight the new features that were considered by "experts" to be important, groundbreaking or otherwise worth a comment (of course, the full feature list would be linked to as well).

I'm not arguing this policy is perfect, nor that it doesn't break down in the scenario you cite, but it does have a solid basis.

One other policy is that Wikipedia does not have firm rules (for this very reason) and so you could say that making a convincing argument such as you have should keep the material out. In principle this works, in practice it doesn't but only because of the community dynamics (a whole other problem!).

while it means there is some possibility that the information was gathered in a way that we should indirectly trust, as we can't see it we can't verify it, and it is honestly not in any way better than if some random person on Wikipedia just asserted it to us...

To an extent it does. Because the reporter who you cite has his/her real name attached to the article and has a public reputation to uphold.

Re:Wikipedia Is Nearing Completion, in a Sense by saurik on Monday October 29, 2012 @09:42PM (#4715611)
> ...ensuring that there is some limit on the scope of Wikipedia. It is this latter one that is the key facet.

Right... but in a world where I can store the entirety of Wikipedia on my mobile phone, you have to trabsitively ask why there is a need a "limit on the scope of Wikipedia". I see no a priori reason why Wikipedia needs or even should tolerate such limits, so one must examine te arguments used to defend that policy.

So far, the only reasonable arguments I have heard (as in, discounting technology problems that never existed: you can easily scale Wikipedia to have a bunch of mostly-ignored articles) come down to "verifiability" through the argument path I elaborated (and which britta seeded), and that is precisely the path used by people defending "deletionism" on behalf of Wikipedia editors.

Re:Wikipedia Is Nearing Completion, in a Sense by ErrantX on Tuesday October 30, 2012 @06:06AM (#4716774)
There are a number of good arguments.

Where does the scope of Wikipedia end? Should there be an article about "saurik"?

How do you actively police articles for e.g. defamation (note, we already struggle to handle this problem and it is getting worse)?

How do you stop spam?

I like to come at this argument from the opposite direction: what need is there for Wikipedia to give an article to every single trivial thing. Is what the president had for breakfast in 2011 sufficiently interesting to the reader?

Wikipedia is not a dump of knowledge, it is supposed to be a curated summary of the sum of human knowledge. And as with an article where you make editorial decisions about the level of detail to go into, so the entire Wiki is scoped to a reasonable level of detail.

Re:Wikipedia Is Nearing Completion, in a Sense by saurik on Monday October 29, 2012 @09:01AM (#4712021)
> Interesting idea, and I'd like to hear more of this idea!

I managed to dig up one of the examples: IBM's History Flow. I believe it was in a course in Cognitive Science that I was taking back in 2004 where I first heard of this research.

http://alumni.media.mit.edu/~fviegas/papers/history_flow.pdf

I feel like there was something similar that was much lighter-weight (as in, not so "large") that I had seen presented at one point, but I had a hard time finding even this one. :(

Re:Wikipedia Is Nearing Completion, in a Sense by mjn on Monday October 29, 2012 @07:56AM (#4711848)
The newspaper problem I agree is a problem. I would generally not consider newspapers to be "secondary sources" (or "reliable sources"), though, but actually something closer to primary sources. Historians typically treat them as primary sources, one source of contemporary information that needs to be carefully interpreted in order to determine what happened. But for recent events they might be the best that's available.

For example, Wikipedia's articles on World War II should be mainly based on existing WW2 scholarship, not on Wikipedians individually doing research into newspaper or military archives (many of which are now online so this is easy to do), because assessing contemporary newspaper accounts of WW2 for reliability is a difficult job, and Wikipedia isn't the place to peer-review attempts to do so when there is already so much good historical work on the subject, and better-suited venues to vet proposed revisions to the historical consensus. So Wikipedia's job should be to summarize the current historical literature on the subject, not to attempt a novel historical synthesis from primary sources.

With scientific articles it's a bit trickier. Individual articles are citeable, but I typically prefer meta-analyses or review papers. Since the goal of Wikipedia is to summarize existing consensus in a field (while also documenting any significant divergent views), those are the strongest indicators. An individual article sometimes indicates the existing state of scientific knowledge, but has to be used carefully, because it might already have been refuted by another article, or it might be a proposal that, while considered good enough to pass peer-review and be worth publishing, is so far only accepted as conclusive by a small minority of the relevant scientific community. (Some journals publish commentary/responses along with controversial articles, which can be useful to help determine if that's the case.) I guess that's also true in history: just because Speculum published an article where someone proposes a large-scale revision to our understanding of 13th-century France doesn't mean that medievalists generally agree with the new proposal. So if it's a recent article and there isn't indication of it having been taken up into the new consensus, I'd cite it in support of a "but some historians argue..." statement rather than in support of an unqualified statement.

This rule also, pragmatically, avoids re-litigating whole areas, like fringe physics and holocaust denial. Maybe someone has great original arguments for their physics theory, or for why the holocaust never happened, but because of "NoR", Wikipedia isn't the place to make those arguments.

Re:Wikipedia Is Nearing Completion, in a Sense by saurik on Monday October 29, 2012 @08:07AM (#4711875)
I am not in any way arguing that people should do "original research"; can you point out where I have done so?

You also seem to be taking a non-Wikipedia view of the definitions of these various varieties of "source".

o Wikipedia makes it clear that articles should be based on secondary sources, using primary sources only sparingly and preferably only when backed up by secondary sources.

o Wikipedia makes it clear that research in a field, such as the results of an experiment or the proof of a theorem in a field such as mathematics, is a primary source.

o Wikipedia makes it clear that for their purposes a newspaper article is a secondary source (not a primary one), and in fact is an example of "the most reliable sources" available.

o Wikipedia also, at least, makes it clear that a summary article written in an academic journal is a secondary source. This is really not that bad, but I demonstrate in another thread how summary articles are a painful limitation.

Yes: in the field of history, these are defined differently, and even Wikipedia drops their notion of "secondary source" and instead comes up with something different. However, I am not talking about scrounging up ancient newspapers when I say "primary source": I mean either bypassing the error-laden filter of the New York Times or citing the actual experiments when discussing those experiments, rather than citing summaries or even journalism.

Re:Wikipedia Is Nearing Completion, in a Sense by derleth on Sunday October 28, 2012 @09:06PM (#4710574)
It isn't about explaining it better, it's about the fact people have a need to feel special and some people peg that specialness to having an article about Their Thing on Wikipedia. Patiently explaining to them that, without external verification, there's no way to prevent them from simply making shit up is not going to reduce their sense of indignation when their unverifiable stuff is deleted.
Re:Wikipedia Is Nearing Completion, in a Sense by saurik on Monday October 29, 2012 @04:28AM (#4711491)
The way these rules are currently set up does not solve this problem: you simply "make shit up" when talking to reporters and then let people reference the secondary source (which is usually designed to fact check information by simply calling you and asking "was this true", at which point you can easily just lie; more information on this in my upstream reply).

You simply cannot use secondary sources to solve the problem you are describing, and if you tried to do so in an academic article you would find yourself unable to get published: you would simply find a note attached from peer review asking you to instead cite primary sources... as otherwise there is nothing to prevent you from "simply making shit up".