Not any more than you worry about someone kicking in your shit-tacular cheap door, or drilling the lock out. Homes aren't often fortresses; you'd build differently if you lived in an unsafe, unstable area. Lots of homes in America have no property walls or fences, either--probably the supermajority of them. I assume that's also true for Canada, especially in uh, the woods.