One question: how do you detect when a binary is intentionally made to statically look similar to one binary, while its behavior actually mimics another?
In my work I'm not looking for intentional spoofing, but the obfuscations I do use [3,4,5,6,7] end up building very similar control flow structures for different functions. Maybe that fits the spirit of your question... Let me know if not.
So far I'm doing purely static analysis and control flow, but the broader field of reverse engineering includes dynamic/symbolic analysis where you track values through a running/simulated program. Great results but very costly to run.
I've been focusing on making cheap/static analysis better, so I haven't explored the dynamic/symbolic side at all yet.
[1]: https://tigress.wtf/virtualize.html
[2]: https://tigress.wtf/jitter.html
[3]: https://tigress.wtf/flatten.html
[4]: https://tigress.wtf/split.html
[5]: https://tigress.wtf/merge.html
My approach is basically: train a model on a large obfuscated dataset seeded with "known" examples. While you can't say with certainty what an unknown sample contains, you can determine how similar it is to a known sample, so you can spend more of your time analyzing the really weird stuff.
The hardest part in my opinion is generating the training data. You need a good source code obfuscator for your language. I've seen a lot of papers that use obfuscator-llvm[1] to obfuscate the IR during compilation. I use Tigress[2] to obfuscate the source code because it provides more diversity, but it only supports C.
[1]: https://github.com/obfuscator-llvm/obfuscator/wiki/Installat...
[2]: https://tigress.wtf/