News for hackers. Stuff that matters.
Re:Visualizing Large-Scale and High-Dimensional Data by cs702 on Sunday May 08, 2016 @08:34PM (#11656802)
I've just skimmed the paper, and this new algorithm looks very nice -- the authors call it "LargeVis."

According to the paper, LargeVis improves on Barnes-Hut t-SNE in two ways: first, it uses the idea that "the neighbors of my neighbors are likely my neighbors too" to construct an approximate graph of nearest neighbors in the high-dimensional space in a manner that is computationally much more efficiently than the method used by t-SNE. Second, the authors apparently have found a clever way to use SGD to map this graph to two (or three) dimensions with computational cost linear in the number of nodes.

If the authors release an open-source implementation, LargeVis looks likely to supplant t-SNE as the go-to algorithm for visualizing high-dimensional data.

Re:Visualizing Large-Scale and High-Dimensional Data by eugene_ducker on Monday May 09, 2016 @01:58AM (#11657870)
The idea the neighbors of my neighbors are likely to be neighbors is basically an assumption of positive curvature. In large codimension embdedded submanifolds can have very negative curvature, in which case neighbors of neighbors might not be as likely to be neighbors as one might first think.
Re:Visualizing Large-Scale and High-Dimensional Data by emn13 on Monday May 09, 2016 @06:36AM (#11658741)
I don't think that's a serious restriction.

Firstly - in general it's trivially true that you have probability 0 to cleanly embed a high dimensional space into a 2/3 dimensional representation over the set of all possible high dimensional data - yet interesting data often does have lower-dimensional structure.

Secondly - so what? Can you think of plausible scenario where this assumption does not hold and it's possible to generate a low-dimensional embedding? If it's impossible to embed, then it's not an algorithmic problem if you fail to find an embedding.

Re:Visualizing Large-Scale and High-Dimensional Data by cs702 on Monday May 09, 2016 @07:43AM (#11659014)
Correct. The authors in fact mention this in the paper, and state that this is probably not an issue because in practice most large high-dimensional datasets tend to have points which lie on/near embedded submanifolds of much lower dimension.
Re:Visualizing Large-Scale and High-Dimensional Data by vladimirralev on Sunday May 08, 2016 @09:21PM (#11656964)
I wonder why they are positioning this as visualisation technique rather than dimensionality reduction which would be a much bigger deal. Is there something to suggest the technique doesn't work well for reduction in arbitrary dimensions?
Re:Visualizing Large-Scale and High-Dimensional Data by eximius on Sunday May 08, 2016 @09:50PM (#11657079)
Disclaimer: Havent read it but have read summaries here.

It sounds like it is very fast but not very rigorous. This lets you get a feel for the data but it doesn't give you the same guarentees other dimensionality reductions do.

Re:Visualizing Large-Scale and High-Dimensional Data by cs702 on Sunday May 08, 2016 @10:36PM (#11657262)
That's my sense from skimming the paper.
Re:Visualizing Large-Scale and High-Dimensional Data by emn13 on Monday May 09, 2016 @06:36AM (#11658745)
What kind of guarantees are you thinking of?
Re:Visualizing Large-Scale and High-Dimensional Data by visarga on Monday May 09, 2016 @12:15AM (#11657580)
I read a paper [1] about visualizing the Deep Q-Network used for playing Atari games. They map to 2D the game state and then colorize it by the Q scores. As a result they could visualize the strategy and observe how it is hierarchically organized in clusters containing sub-clusters. They could show regions associated with initial and termination rules. This method can be used to map out a game strategy and to focus on specific regions.

[1] Graying the black box: Understanding DQNs - https://arxiv.org/pdf/1602.02658.pdf

Re:Visualizing Large-Scale and High-Dimensional Data by cbsmith on Wednesday May 11, 2016 @09:41AM (#11675134)
The dimensionality reduction problem space rarely tries to reduce down to only two or three dimensions. The fact that in the end you are being reduced to so few dimensions allows for a lot of potential errors to not meaningfully impact the result.
Re:Visualizing Large-Scale and High-Dimensional Data by st553 on Monday May 09, 2016 @11:21AM (#11660481)
Anyone know if its possible or makes sense to use something like t-sne for dimensionality reduction? If so, could the reduced data set be used to build a classifier?