I also agree that modeling the time dimension by MLP can be more rational than self-attn. It learns the weightings from time points with the same physical meaning.