The UI part will be in charge of converting plaintext into ciphertext and vice versa. ciphertext will be handed off to the transport module.
The transport module can remain closed source. Only the API to the transport needs to be published. People can write their own UIs.
The biggest problem with all of it was that there were a bunch of scam sites that added malware, built binaries, and bought "lime wire" keywords on google.
On the other hand, I don't think OSS is to blame for that--the scam sites could have just as easily distributed any binary.
This is where the distinction between "free" (as in freedom) and "open source" is helpful.
You can, hypothetically, release the source code of a project under a license that prohibits compilation of that source code (or, prohibits running anything other than the paid binary of the source code). This would allow people to view and theoretically vet the code; they just can run it (legally) without paying for it.
Not that I would like to encourage such behavior, or think that it's valuable. But it's an important distinction to remember.
Such a license would qualify for neither "open source" nor "free software" under the relevant official definitions though.
Yes, it would be reviewable for bugs and probably preferrable to a blob. But without the ability to verify the complication you'd have no assurance that the proprietary code was actually built with the reviewed source. Basically this would just be a stunt.
I recall Transgaming Wine had a model that was effectively this, it was difficult for a laymen to build the source and binaries couldn't be distributed freely but the source was still available.
That is "source visible", I guess. But please don't confuse terminology: it's neither "open source" nor "free software".
That doesn't invalidate the point above though that in the modern world a tool like this can only be considered "secure" if the implementation(s) are completely open. It's just a poor product decision on the part of BitTorrent.