Yea I wish it was possible. The read/write repo permission is the only way I've found to be able to list all of the private repositories in an org. There's a few other threads in these comments that cover the specifics a little bit better but the gist is that GitHub has pretty coarse grained permissions unfortunately.