If you don't want to try to claim that land through adverse possession you should look into getting an easement/covenant drafted and on file for perpetuity so when you move out the next owner can't turn around and hassle your friendly (ex-)neighbors with the threat of it. (Or if your neighbor moves out, the new owner wont be able to evict your swingset.)