Beginner here. Having done most of the Karpathy YT course I have seen embeddings trained as part of a larger net that is predicting the next token. After the embedding is run on several tokens the output goes though several layers of linear;batchnorm;nonlinear. I haven’t done the wavenet or transformer stuff yet.

How can you train the embedding without that? And if you can, is it a good or bad idea to prepare your embedding using this script first then chuck those weights into a bigger model (I think I know what the answer will be…)