I believe the right approach will begin with something like - 1. use an autoencoder to encode the momentary state. 2. use an autoencoder to encode a temporal snippet of let's say 5-10s. 2. Use that as the input tokens for a LLM.