I saw something similar before (from gifs.com's sticker editor of all things lol) where you annotate the segmentation of the first frame of the video and it will propagate that segmentation to the rest of the frames:

https://medium.com/gifs-ai/interactive-segmentation-with-con...