Fortunately, that entity sits above California's voters.
Except the population votes for the people who pick the members of that entity.
We've added a few hops, and hopefully increased the average intelligence of the people who get to decide on rights, but they still end up there based on the population's vote.
Since we don't live in that ideal world, our only option is to designate some body to decide what are rights and what are not. Whether that body is a king or a judge or a council of judges or a legislative body or the full populace (or some combination of the above) is an important and worthwhile discussion, but in the end we do need to designate some entity to decide what the rights are.