People have actually tried to do things like this; though I suspect a linear transformation wouldn't work well. Chris Olah talks about it in his awesome blog: https://colah.github.io/posts/2014-07-NLP-RNNs-Representatio...