It doesnt matter if server code is proprietary if E2E keys never leave device. And open source server change nothing since you cant verify what code is running on actual production server anyway.

And messages that are not E2E encrypted are unsafe by default.