It's becoming downright common for Uber drivers to outright admit they give 1 star ratings for non-cash-tippers.
But yes, I've anecdotally heard of ratings drop for women who typically are type-A chatty, but changed to quiet and withdrawn for a few weeks during major life events. Perhaps that came across as more surely, but just something as silly as someone perceiving a bad attitude from a rider can results in a lower rating score.
I've also talked to drivers who give poor ratings if they have to go well out of their way, and it ends up being a short ride. Sure, this really sucks for the driver - but is it the rider's fault at all? Other unpopular destinations get similar treatment (or so I've heard).
Maybe Uber shouldn't be making any decisions about users. Just present the rating to the driver pool and let drivers decide who they pick up and who they do not. It doesn't entirely solve the issue of "what is a justifiable reason for rating a rider (or driver) poorly?" but it makes it less catastrophic for users.