News for hackers. Stuff that matters.
Re:Wikipedia Is Nearing Completion, in a Sense by ErrantX on Monday October 29, 2012 @06:32AM (#4711696)
Worse, as encyclopedias--which today pretty much means "Wikipedia" to a very large percentage of the public--are used as reference material for the construction of secondary sources, everything from newspaper articles to books, claiming secondary sources to have anything to do with "validation" just makes your chain of evidence circular.

This is a phenomen Wikipedians are aware of. But what you refer to here are often tertiary sources, and not necessarily the sort of material you'd want to use.

The result of attempting to find truth from secondary sources is that you will forever be plagued by horrible bias, both in terms of what things you can find to be "notable" and in terms of the validity of the opinions you can find in them.

You're arguing to swap that bias for your own? i.e. you look at primary source material and decide which items are notable for their own article. What about if another editor disagrees?

In academia secondary sources are an established way of processing primary information; an expert reviews the primary material and, citing it, draws conclusions. Other experts may do the same, and disagree with the first one.

Wikipedia is a tertiary source, which draws on secondary and, yes, primary material to summarise the current status of a topic.

However, the article in the New York Times quite clearly was entirely itself sourced by simply asking the people who worked for Diggnation and Revision3 a bunch of questions; it even quite clearly states that that number came from them: "Revision3 says it counts roughly 250,000 views each week".

In such a situation, the way the New York Times operates (and I know this first hand, as there have been articles about the things that I do published by them) is that they will happily publish whatever number you tell them, and at best "fact check" it by calling you back to verify they heard you correctly.

This is a problem, and one with no easy solution. The approach of requiring sources to have their own review process (i.e. media has editors who theoretically vet content) should fix this, but as you point out, in practice it is not 100% reliable.

Re:Wikipedia Is Nearing Completion, in a Sense by saurik on Monday October 29, 2012 @07:14AM (#4711775)
> You're arguing to swap that bias for your own? i.e. you look at primary source material and decide which items are notable for their own article. What about if another editor disagrees?

treelovinhippie's stated opinion was "remove the "must be noteworthy" rule" (from which I am mentally rewriting his statement to "notability"). I personally agree with that, as I do not believe that the notability requirement is leading to increased article trustworthiness.

I would rather see the idea of "notability" replaced with an "article score" based upon the history: there have been a few glorious visualizers for Wikipedia designed to really make this information hit center, and I think any of them would be a better global solution than "notability".

> In academia secondary sources are an established way of processing primary information; an expert reviews the primary material and, citing it, draws conclusions. Other experts may do the same, and disagree with the first one.

I imagine this is determined heavily by the field? I can't imagine in either mathematics or algorithmic computer science there being any need to wait until a review paper is published "drawing conclusions" (whatever that would mean) from the information in the primary research.

However, I honestly feel like you are dragging me down an academic rathole in these very specific cases, such as medicine: the kind of material that we are explicitly discussing here, Diggnation, is not going to be covered in anything remotely approaching a scholarly source.

Instead, what we are comparing are situations like 1) referencing a New York Times article stating that "person Y said X" and 2) directly linking to the blog or Twitter feed of person Y and demonstrating first-hand that they said "X": in these situations, removing a layer of required trust.

In my experience discussing situations like this with Wikipedia editors, in addition to my direct attempts to read through the rules that Wikipedia publishes for how their website should be used, it is quite clear that Wikipedia would prefer you to cite the NYT article rather than hotlink the information.

> This is a problem, and one with no easy solution. The approach of requiring sources to have their own review process (i.e. media has editors who theoretically vet content) should fix this, but as you point out, in practice it is not 100% reliable.

So, Wikipedia specifically states that "mainstream newspapers" are an example of "the most reliable sources". Can you provide me a reference to where Wikipedia is claiming that you can only depend on sources that "have their own review process"? (Or, was that just a potential idea for a fix?)

Personally, in this situation, I would much prefer a reality where Wikipedia simply admits that secondary sources (in this field; again: not medicine, and possibly not even history) are nothing more than opinions (in the general case, this is how Wikipedia defines secondary sources: opinions and interpretation).

If they did this, then they couldn't cite a New York Times article for a sentence that simply stated a fact: they could only cite a New York Times article and state "the New York Times states", for which the New York Times should be a perfectly valid (and I will argue "authoritative") reference.

Yes: in this case that would probably make it impossible to just state "Diggnation has 250,000 subscribers" without some kind of in-article qualification; however, I don't think that that is inherently a bad thing: I feel like if I were writing a scholarly article on this subject, I wouldn't be able to say anything definitive either.

Re:Wikipedia Is Nearing Completion, in a Sense by ErrantX on Monday October 29, 2012 @07:34AM (#4711810)
I personally agree with that, as I do not believe that the notability requirement is leading to increased article trustworthiness.

I struggle to follow this because Wikipedia doesn't consider "notability" to relate to article trustworthiness. It's instead intended to act as a soft margin for the topics that deserve a standalone article (remember, noteworthiness is only related to whether an article should exist or not).

I would rather see the idea of "notability" replaced with an "article score" based upon the history

Interesting idea, and I'd like to hear more of this idea! However, from this I suspect you are mixing up the reliability of an article and the specific phrase "notability" (about how deserving a topic is to have an article). When deciding if an article is notable it will usually have little or no history!

I imagine this is determined heavily by the field?

Yes, a point Wikipedia admits.

I can't imagine in either mathematics or algorithmic computer science there being any need to wait until a review paper is published "drawing conclusions" (whatever that would mean) from the information in the primary research.

Then you would, I am afraid, be wrong :) In such a field many papers are published contending a theory, and then other experts review the contention and submit responses/reviews/criticism.

the kind of material that we are explicitly discussing here, Diggnation, is not going to be covered in anything remotely approaching a scholarly source.

Yes, which is the key to the problem.

Can you provide me a reference to where Wikipedia is claiming that you can only depend on sources that "have their own review process"?

Yes, the core sourcing policy requires a "reliable source" (http://en.wikipedia.org/wiki/Wikipedia:RS). "Articles should be based on reliable, third-party, published sources with a reputation for fact-checking and accuracy". http://en.wikipedia.org/wiki/Wikipedia:SOURCES#Reliable_sour... goes into more detail.

Instead, what we are comparing are situations like 1) referencing a New York Times article stating that "person Y said X" and 2) directly linking to the blog or Twitter feed of person Y and demonstrating first-hand that they said "X": in these situations, removing a layer of required trust.

Is it? Now I agree that if the NYT has simply asked Diggnation for those figures then Wikipedia should note that is where it comes from. However citing it to NYT is intended to demonstrate that someone other than the editor inserting the material trusts its veracity. Its difficult because we simply do not know where NYT got that info: if Diggnation showed them some real figures (say, a screenshot) then the NYT article is better than a tweet!

Which is why this is not a simple problem.

Re:Wikipedia Is Nearing Completion, in a Sense by saurik on Monday October 29, 2012 @08:19AM (#4711905)
> I personally agree with that, as I do not believe that the notability requirement is leading to increased article trustworthiness.

> Interesting idea, and I'd like to hear more of this idea! However, from this I suspect you are mixing up the reliability of an article and the specific phrase "notability" (about how deserving a topic is to have an article). When deciding if an article is notable it will usually have little or no history!

The reason I am "mixing this up" is that the only argument I have ever heard for why Wikipedia needs a notability policy at all is the one that was cited by britta earlier in this thread: that without such a clause there would be tons of articles that are hardly ever looked at by anyone, hardly ever edited by anyone, and containing information that is difficult to verify. I believe that there are numerous better solutions to this than attempting to use the "notability" filter, as I maintain that "notability" does not actually lead to "veracity".

To be very clear about this, I will repeat the context from britta that started my involvement in this discussion:

>> The notability guideline (http://en.wikipedia.org/wiki/Wikipedia:Notability) is closely related to the verifiability policy (http://en.wikipedia.org/wiki/Wikipedia:Verifiability) — from Wikipedia's perspective, if a subject has not been covered by multiple reasonably reliable secondary sources (in other words, if it isn't "notable"), we can't write a reasonably verifiable article about that subject. Every article has to include secondary sources as references, so that editors and readers can quickly fact-check.

Going back to your response:

> Its difficult because we simply do not know where NYT got that info: if Diggnation showed them some real figures (say, a screenshot) then the NYT article is better than a tweet!

I must apologize here, as I was intending to be using a different example, but you took me to mean the Diggnation example: this is my fault, I should have been more clear.

What was coming to mind with relation to the "Twitter post" example, is that there is a ton of journalism on topics I directly care about that is based on the Twitter feeds of people I work with: a lot of "person X said Y", which is then translated into some article on what is or is not possible with a tool that person X builds. The fact that a reporter read that statement and repeated it doesn't make it more true, and in fact the opposite is quite common: they are paying sufficiently little attention and have sufficiently little background knowledge that they repeat it wrong.

Please understand: I am not talking about a situation of "interpretation" or "research", but more like establishing dates on when things happened... I understand that this feels pretty blurry (but it seems equally silly to go into a detailed example of something where my example is itself biased; I am happier sticking with the examples such as Diggnation and RSA); it is simply a situation where "that dude at Wired that finds this stuff gets him readers" is not somehow more trustworthy than the place he got the information from... either the information shouldn't be published at all (I believe this is probably a quite reasonable course of action), or "that dude at Wired" should be skipped and the original source should be used.

That said, part of your comment doesn't rely on that misunderstanding I caused: it is true that we don't know where the NYT got that information, however that uncertainty really doesn't make it any more true; while it means there is some possibility that the information was gathered in a way that we should indirectly trust, as we can't see it we can't verify it, and it is honestly not in any way better than if some random person on Wikipedia just asserted it to us... reporters for major publications (such as the New York Times and Washington Post, both of which I have first-party experience with due to articles written about my work) really do trust that you are a reliable source on things that you control, and really do attempt to fact check by calling you back for verification.

Re:Wikipedia Is Nearing Completion, in a Sense by ErrantX on Monday October 29, 2012 @09:40AM (#4712173)
Ok, fair enough. Britta touches on some of the issues, but not in its entirety. Notability is about a) requiring there be at least one reliable third party source (so that the article has a chance of containing verifiable information) and b) ensuring that there is some limit on the scope of Wikipedia. It is this latter one that is the key facet.

Whilst Notability is closely related to Verifiability it is not quite in the way britta cast it, but rather related to requiring the material used to define a subject notable (i.e. a significant claim to importance) is verifiable. i.e. the relationship works the opposite way.

I was intending to be using a different example, but you took me to mean the Diggnation example: this is my fault, I should have been more clear

Ah, my apologies, I'm reading quickly as it is a busy day.

Please don't get me wrong; the issue you highlight is a major problem, one I have raised a few times internally with the community. But there has been no easy resolution.

It's worth noting that the reliable sources policy explicitly notes that reliability hinges on not only the publisher but also the content and the author. If an author is seen to lack the qualifications, or has a bad reputation, these factor into consideration.

With that said; a lot of Wikipedians don't know this. A problem I run into constantly when discussing sources ("Well, it was published by the NYT's so it doesn't matter what their reputation is"). It's not the policy at fault there, but the lack of interest of our own community in the rules...

"that dude at Wired that finds this stuff gets him readers" is not somehow more trustworthy than the place he got the information from...

The intent of the policies (and bear in mind what I say above as to how much that holds up..) is that the secondary source is used to filter what in the primary material is considered important to the wider community. To take an example: when Microsoft released Windows 8 there was quite an extensive list of new features. Simply recording that isn't what Wikipedia aims to do, instead you would use secondary sources to highlight the new features that were considered by "experts" to be important, groundbreaking or otherwise worth a comment (of course, the full feature list would be linked to as well).

I'm not arguing this policy is perfect, nor that it doesn't break down in the scenario you cite, but it does have a solid basis.

One other policy is that Wikipedia does not have firm rules (for this very reason) and so you could say that making a convincing argument such as you have should keep the material out. In principle this works, in practice it doesn't but only because of the community dynamics (a whole other problem!).

while it means there is some possibility that the information was gathered in a way that we should indirectly trust, as we can't see it we can't verify it, and it is honestly not in any way better than if some random person on Wikipedia just asserted it to us...

To an extent it does. Because the reporter who you cite has his/her real name attached to the article and has a public reputation to uphold.

Re:Wikipedia Is Nearing Completion, in a Sense by saurik on Monday October 29, 2012 @09:42PM (#4715611)
> ...ensuring that there is some limit on the scope of Wikipedia. It is this latter one that is the key facet.

Right... but in a world where I can store the entirety of Wikipedia on my mobile phone, you have to trabsitively ask why there is a need a "limit on the scope of Wikipedia". I see no a priori reason why Wikipedia needs or even should tolerate such limits, so one must examine te arguments used to defend that policy.

So far, the only reasonable arguments I have heard (as in, discounting technology problems that never existed: you can easily scale Wikipedia to have a bunch of mostly-ignored articles) come down to "verifiability" through the argument path I elaborated (and which britta seeded), and that is precisely the path used by people defending "deletionism" on behalf of Wikipedia editors.

Re:Wikipedia Is Nearing Completion, in a Sense by ErrantX on Tuesday October 30, 2012 @06:06AM (#4716774)
There are a number of good arguments.

Where does the scope of Wikipedia end? Should there be an article about "saurik"?

How do you actively police articles for e.g. defamation (note, we already struggle to handle this problem and it is getting worse)?

How do you stop spam?

I like to come at this argument from the opposite direction: what need is there for Wikipedia to give an article to every single trivial thing. Is what the president had for breakfast in 2011 sufficiently interesting to the reader?

Wikipedia is not a dump of knowledge, it is supposed to be a curated summary of the sum of human knowledge. And as with an article where you make editorial decisions about the level of detail to go into, so the entire Wiki is scoped to a reasonable level of detail.

Re:Wikipedia Is Nearing Completion, in a Sense by saurik on Monday October 29, 2012 @09:01AM (#4712021)
> Interesting idea, and I'd like to hear more of this idea!

I managed to dig up one of the examples: IBM's History Flow. I believe it was in a course in Cognitive Science that I was taking back in 2004 where I first heard of this research.

http://alumni.media.mit.edu/~fviegas/papers/history_flow.pdf

I feel like there was something similar that was much lighter-weight (as in, not so "large") that I had seen presented at one point, but I had a hard time finding even this one. :(