Fact: Bromine is an essential element in mammalian biology. We need to consume it.

Fact: Bromine does not preferentially crowd out iodine. It does tend to concentrate in the same tissues as iodine for the obvious chemistry reason but it does not materially impede iodine absorption. Which has actually been measured in mammals in biomedical studies.

Fact: Some populations, like the Japanese, consume relatively huge quantities of bromine from natural dietary sources while still having (modestly) lower levels of thyroid disease than cultures that consume much less. This is an expected demographic result given the aforementioned biomedical research that would predict this, given that bromine does not crowd out iodine.

If we biologically require both iodine and bromine, which we do, it is pretty reasonable to assume that evolution makes sure the ions get to where they need to go. Human biology is full of superficially conflicting ion requirements: iodine and bromine, sodium and potassium, calcium and magnesium, etc.

As an added bonus, ions like bromine (and iodine) tend to be promptly excreted if you consume more than your body actually requires, so they don't stay in your system long in any case if they are not serving a purpose. There is nothing intrinsically dangerous about bromine.