Yes. We are using rolling sentences. Not just pairs. But the whole story. I have written a little about this in my previous post, shape of stories, https://gpt3experiments.substack.com/p/the-shape-of-stories-...

So we can see how as we are appending sentences and getting the embedding we get the movement of the story in the latent space.