I used the same setup and was also frustrated. I replaced Orgzly with Orgro [0] which can only view files but works much better for my purposes.

[0] https://github.com/amake/orgro/