News for hackers. Stuff that matters.
Re:Visualising Correlations using Graphs by taliesinb on Friday December 05, 2014 @02:07AM (#8703291)
The most well-known practitioners of this sort of 'topological' approach are Ayasdi, they have some slick demos [1]. The general name for this idea is topological data analysis [2].

I replicated this particular experiment in WL, of course, because it's a 5-minute thing to do [3], and I could actually do the community detection the author alluded to.

But I noticed that the correlation matrix itself is much more suggestive than the graph ends up being, with or without community detection. Take a look at the correlation matrix (note that MatrixPlot does some clever combination of rank and absolute value to get high dynamic range):

http://imgur.com/WKn029o

The tri-diagonal structure is because the original dataset is derived from the pixel counts from successive 4x4 tiles on NIST written-digit images [4].

Those 8x8 matrix of tiles is flattened onto the 64 random variables, so the large correlation with tiles on the left and right explain the 1-off-diagonal orange lines, the other two diagonals are offset by 8 and correspond the high correlation with the tiles above and below. That's the 'connectivity kernel' of a 2D manifold, so to speak.

The curious squiggles in all the other blocks of this matrix are unusual. I don't know what's going on there. Maybe something interesting.

[1] http://www.ayasdi.com/

[2] http://en.wikipedia.org/wiki/Topological_data_analysis

[3] https://www.wolframcloud.com/objects/c7927909-448d-4502-9c1a...

[4] https://archive.ics.uci.edu/ml/machine-learning-databases/op...

Re:Visualising Correlations using Graphs by jburroni on Friday December 05, 2014 @09:42AM (#8704712)
Thank you for your feedback. I din't know that topological data analysis even exist! I have some published and unpublished research on the topic. I had used this particular dataset by random. I wanted to use something with integers, large sample size and with high dimension.

I'm glad you did the community detection stuff. I found it useful when the dimension is really large (in the order of thousand variables). For instance, when doing text mining. very nice comment!

Re:Visualising Correlations using Graphs by taliesinb on Friday December 05, 2014 @02:36AM (#8703333)
It turns out that the graph is a bit more interesting when you can understand what part of the original image each node comes from. I've color coded each variable, along with a little legend image:

http://imgur.com/TeOspAj

Re:Visualising Correlations using Graphs by jburroni on Friday December 05, 2014 @09:42AM (#8704719)
This idea is actually very interesting!