DirectShow (the API for video capture and other things in Windows) has long had the concept of a filter which can be plugged into the video pipeline[0]. I'm not certain if that's the reason that the term is commonly used for effects such as the one discussed here, but this "cat filter" certainly might have been implemented as a DirectShow filter, so it's very plausible the terminology comes from that.

I'm pretty sure apps like this were called filters long before Instagram even existed.

[0] https://docs.microsoft.com/en-us/windows/win32/directshow/di...

I thought it started in Snapchat, where they were just filters at first, but then the AR stuff started getting added and you’d access them the same way you would the plain filters. The name just stuck due to people not caring to call them something else. What else would they be called? Lenses, maybe?