You (a) don't have a Chrome extension that verifies anything right now, (b) are currently publishing the pure- browser- JS version of your application, and (c) envision a future in which a small number of "power users" can "safely" use the system while the majority of users, who you correctly note will not install your extension, will be at the mercy of attackers or, for that matter, your own server.

You're essentially asking to be the next Lavabit. You've added a verification component that probably won't work even for the verifiers. But that doesn't even matter, because the majority of your users won't even be doing that verification.