I am not sure what you are doing, but can you just compute the similarity between two frames, and analyze only the novel frames?

I.e. I think that in one minute video, 95% of your images do not have new information in them