The vapor is toxic, one of my college professors said his graduate adviser was suffering from years of exposure to mercury fumes because in the bad old days they used to seal the gas valves in the lab with a little pool of mercury. Poor guy spent 40 years in the lab breathing mercury fumes.
It's not too weird if you think about their physical properties.
Metallic mercury - not that reactive or soluble in water; in fact eating metallic mercury is not that dangerous, it tends to just pass through your body
Mercury salts - Mercury(II) salts are water soluble, but Hg(II) reacts with thiols pretty rapidly (it's chelated by them). That includes all of the sulfur containing amino acids in your body (cysteine and methionine). The problem is it will also bind to important enzymes. Hg(II) salts with trash your kidneys, but don't make it into the brain.
Organic mercury - fat soluble, can penetrate most membranes of the body; that means it gets into the brain and through metabolic transformation results in neuron death