Yes, video is processed frame by frame. Video is essentially just a series of images played back at a specific rate. Different formats & protocols work on how to reduce duplicate sections of the frames, but overall processing is always frame by frame. Given that, there are a number of shortcuts you can apply to frames where you don't need to process each frame 100% separately, but that's more an optimization as you are still processing frames.