Yeah, I'd been working with them for ages before I really wrapped my head around the sausage machine. There was a Wolfram blog a while back that really helped me grok it [0]. The last state of a neural network prior to prediction, if I understand correctly.
[0] https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
The magic is creating the embedding, embeddings themselves are nothing special.