Normally to other plants, rarely for animals (see: poison ivy, poison oak and poison sumac). In really extreme cases can be very dangerous even to burn it (but nobody has a manchineel in their garden). Walnut can poison the soil for a couple of years so is better put its fallen leaves on a isolated compost pile. It depends on the species and the chemistry of the soil.