The previous owner needed a level place for a swingset for his kids, so the next-door neighbor let him use a bit of the neighbor's land. They built the fence so the swingset was on my side of the fence, even though it's the neighbor's land. The fence doesn't change anything. (Though I suppose if I wanted to be nasty, I could claim squatter's rights or some such legal concept, and maybe take that bit. But I don't want to. Ripping off people that were nice to someone isn't how I want to live.)

Some people like to have a fenced space, especially in their back yard (no neighborhood dogs running up to the small kids). Some don't (extended sight lines, more of a feeling of neighborliness). But the fences don't matter. The property lines are still where they are, even if you can't see them.