Yes, this is definitely possible. You can maybe try computing some kind of image distance between frames or some keyframe extraction.

Once you compute the features, the search is very efficient! I tried it for searching in the 2M photos dataset from Unsplash and it takes like 2-3 seconds: https://github.com/haltakov/natural-language-image-search

I plan to run my personal photos through it :)