Simple example: I'm sure that whatsapp main window is webview. Imagine that application inserts some kind of resource (e.g. CSS) from whatsapp server. So now whatsapp server can serve slightly altered CSS which will leak secret data via custom fonts, etc and you won't be able to find that, unless you're intercepting all traffic and can decrypt it (and apps nowadays love to pin certificates).
This is imaginary attack, I have no idea whether whatsapp does that. But HTML is a powerful and dangerous beast, yet it's used a lot in applications for rich media.
Signal has the same issue.
Active ways to attack the client to make it leak the key are far more worrying - but even an open source project wouldn't protect against that.
Good luck finding even this without a fine comb. And that's us just getting started with code flow obfuscation.
No source = no trust. It's as simple as that.
No, it's not "possible enough" and I strongly suspect you fully realize that.
A backdoor doesn't need to be in a form of an IF statement or something comparably obvious and silly. It can be a weakly seeded PRNG that would allow a "determined party" to brute-force the key exchange in a reasonable time. That would take man-years to fish out from a binary, and that's without considering that you may (be forced to) distribute an altered binary on demand and to specific targets only.
So in the end all we have - realistically - is in fact just your word. There's no way for you to prove that you are trustworthy by pointing at some random binary. The only option is to distribute reproducible builds from an audited open source.