bag of words style representations actually can work quite well in many document classification tasks.

wouldn't surprise me at all if affect bleeds through word choice and indicates some emotions like uncertainty or overconfidence that are frequently associated with irreplicable results.

but yes, predictive models of any kind should not take the place of more rigorous validation of past results.