Yes this is really interesting. I haven't read those other papers yet (definitely plan on it now thanks for the links), but Bengio's latest paper on denoising autoencoders from earlier this year (http://arxiv.org/abs/1305.6663) still used the unsupervised pretraining. Also the Theano implementation that I run experiments with uses it as well (but that code could be a year or two old).

Definitely going to be researching this more throughout the year.