There's something called "attentional neural networks" that attempt to do this. They tend to do very well in reading natural language, IIRC, but I've also seen them applied to video.