I've visited libraries in a half dozen countries, where I'm not even a taxpayer. I just walked in.
Some services at the library might not be free, including book checkout or making photocopies.
But if you want to make that distinction, then walking around your neighborhood isn't free because in most place you need to wear clothes, and clothes aren't free.
FWIW, I've walked a lot of forest roads without a car or bike. And the biggest littering problem on our sidewalks, by weight, are the escooter riders who simply leave their machines parked in the middle of the sidewalk.
The problem isn't that the sidewalks are free to use, the problem is one of regulation and enforcement.
That's why libraries are both free and well-loved - they librarians regulate the free space.
Sidewalks are free, but they are usually unpleasant or inaccessible. In places where that isn’t true, they absolutely get trashed by litter; some even spit on the sidewalk.
Libraries are seemingly free, but getting a library card often requires you to provide several forms of proof of address, including evidence of paying rent or property taxes. So they aren’t truly free.