It should be sufficient to inspect the client because end-to-end encryption prevents the server from seeing the message plaintexts. The worst it could do is send the wrong encryption keys to the clients (i.e., attempt a MITM attack, or add unauthorised participants to a group chat), but this can be checked out of band (e.g. QR code in person), and the client provides a message when a contact's public key changes.

If the client implements the Signal protocol correctly, and the key pair is generated securely, private key not transmitted to the servers, etc., then the server should not be able to do anything nefarious without the client noticing.

You can audit the app. Bytecode isn’t hard to read. Pentesters aren’t idiots.