Re:One Year as a Data Scientist at Stack Overflow
by mikk14 on Tuesday June 21, 2016 @07:51AM
(#11945006)
That is a heavily-filtered network: if he didn't drop most of the weak connections he'd end up with everything connected to everything else. On the other hand, if you don't use a sophisticated way to define what a "noisy" edge is, you'll end up with some curious cases like the one you point out. He might have used a naive global threshold -- it's the easiest way to go about it: drop all connections with weight lower than x. But it's also very wrong most of the times :-) Something like the disparity filter [1] works usually well, although you have sure that its null model hypothesis is aligned with what you think is the generative process of your network. The field is "network backboning" and it's a nice one from network science.
[1] https://en.wikipedia.org/wiki/Disparity_filter_algorithm_of_...
Re:One Year as a Data Scientist at Stack Overflow
by var_explained on Tuesday June 21, 2016 @12:55PM
(#11947185)
mikk14 is right; the connections were filtered out below a particular threshold (set that threshold too low and everything is connected to everything; set it too high and you miss meaningful connections).
But note that precisely because git/github is used in combination with almost all other tags, it doesn't have a high correlation with any particular technology. A correlation (roughly) means "If I know you use tag X, you're more likely to also use tag Y." But knowing someone uses git doesn't let you guess what other technologies you use, because as you note they can use almost anything.
In short, if you're connected to everything, that means you're correlated with nothing.
Re:One Year as a Data Scientist at Stack Overflow
by rossipedia on Tuesday June 21, 2016 @10:26AM
(#11945816)
Remember all this data comes from Stack Overflow. Some of the clusters might look a bit odd, but that's because they're all derived from the relationships we've found in Stack Overflow content.
> http://varianceexplained.org/images/network2.jpeg
This shows "git" and "github" in a separate cluster from "C++" and "Python"? I don't understand this. These tools are used regardless of what other technologies are being used. For example, there are many Python and many C++ projects on github.