[1]: https://en.wikipedia.org/wiki/Subgraph_isomorphism_problem
[2]: https://en.wikipedia.org/wiki/K-d_tree
[EDIT]:
Graph canonization [3] on the other hand is not known to be either polynomial time solvable (I think it is for planar graphs though) or NP-complete and has implementations that are quite efficient in practice. It can be used to search for the exact same graph, but again extending to searching for similar graphs is probably quite difficult.
The graph idea could work, but the algorithm produces slightly different results each time it is ran. It still could be really interesting to try!
The compute on the indexing images would be relatively expensive, but the speed of a similarity search would be super fast, with the caveat that I don't know how image search is done today, and it probably should do something like this anyway.
This isn't my area of expertise at all, but intuitively it seems like if you treated the triangles as a graph, you could sample a minimal subgraph from an image and then search for other files that have a similar subgraph. It's conceptually like "geometric hash."
Is that what this technique is for, as it seems to have more applicaitons than just an image filter.