Background levels are much higher than current standards with no observable effects in many parts of the world. This is well-studied. There is a threshold but it is higher than people assume.

Plant-based arsenic often has poor bioavailability. Quite a few plants people eat are natural accumulators but it just passes through. Pesticides and geology are the primary bioavailable sources.