I own a car but I've considered renting cars for trips multiple times (you don't have to care as much about a rental car, you don't mind if other people are driving it, it's normal to split rental car expenses but weird to split, say, the depreciation on your car) etc. But it's often a hassle.
Most rental car centers in the city do not open before say 9 am (the best I have seen is 8 am) and close by 5-6 pm. After you do all the paper work and blah blah it's typically 9-10 am before you can leave for your trip. This is just not practical for a lot of weekend trips. The other option is to rent it on Friday (which means you pay for a day more), but since they close by 5 pm you need to rent it on Friday morning at 9 am which means you have to now plan it along with your work day.
The other option is to rent from the airport where the centers are generally open 24/7 but charge more and the airport is far so now you have to take that into account.
The other problem is when you rent for a long trip and have to pay per day when you know you'll actually be driving only on a few of those days.
* Most US car rental price comparison sites let you book a car months in advance to lock in the price, but you don't pay till pick up. There's almost never a penalty for booking and not picking up, so I used to just block book the weekends months in advance, whether I needed the car or not.
* I just used the big rental firms - at the time services like ZipCar were priced to heavily incentivise you to only drive between ZipCar areas and drop off, e.g. use a different car for the outbound journey and the return one. Using them for all-day tripping often ended up being significantly more expensive than a traditional rental. I've no idea if this has improved. Car sharing services appeared to really rely on both your start and end destinations being in their 'coverage' areas though.
* Once I got to know staff at my local rental place, I started getting 'free' upgrades a lot.
However, after four or five months I ended up buying a modest low mileage 3 year old Honda. The cost of running isn't actually all that much more, and now I don't have to factor in travel time to rental place, having to refill the rental before return, clock-watching etc. This is entirely selfish of me of course, but very much caused by the small 'barrier' you mention.
Hell, I live in Manhattan and I now don't do a lot of things I used to do when I lived out in the suburbs and had a car, for precisely this reason. Instead I do a lot more things on my bicycle, including long weekend trips -- it's basically become to me what my car used to be to me when I lived in the suburbs. Except using it actually improves my health.
The difference between the magnitude of the two ways of stating this "downside" are vast, and I hear people conflate them constantly.
Nothing, but it is a barrier. When you own a car, you just get in, make sure you've got gas, and go. When you don't, you need to go out and find a rental place, get there, find a car you want/like, etc.
It's a small barrier, and mostly psychological, but it's there and it affects decisions.