I get excited whenever we can improve performance with manipulations to the training sets such as data transformation tricks (inducing variances like rotations, translations, whitening [1] etc), labeling tricks (such as those like LeCun or [2]), or including information/learnings/regularizations from external corpus like doc2vec.

It feels like getting something for free :) Of course there is a limit to how much signal you can extract from a noisy dataset, but the amount of time and human energy invested into creating and improving datasets can be quite large relative to finding another cool trick that can improve performance.

However, I wonder which will come first to make these systems "robust" for the average joe's real world uses for these perceptual systems; a large, well labeled dataset or more transformations and semi-supervised learning approaches?

[1] http://www.cs.stanford.edu/~acoates/papers/coatesleeng_aista... [2] http://cseweb.ucsd.edu/~elkan/posonly.pdf