Very interesting, this reminds me of the 2012 paper by Andrew Ng: Building High-level Features Using Large Scale Unsupervised Learning

To further clarify: does unlabeled mean "we didn't use sentiment data" or "we were only trying to predict the next character given the prior characters", since the amazon data does come with associated 1-5 star ratings, were those used or not?

We did not use the star ratings.

Have you looked at cross referencing against the star ratings? Would be interesting to see how predictive this was of those ratings... or even if another variable in the system could predict that?

That's what I thought, and that makes this all the more interesting!