"... open sourcing the server code wouldn't really do that much to improve trust."

It would allow people to run your code (if they like it) on their own servers. Then they wouldn't need to trust you.