From an allergy standpoint small exposure would be enough to reduce risk of developing an allergy.
https://www.nationalacademies.org/our-work/the-role-of-seafo...
Moreover, it has been observed that traditional fishing cultures that have a high mercury intake have zero symptoms of mercury toxicity. The leading hypothesis of this is that the high selenium content in these same fish is protective against mercury exposure[1]. Consider also the observation that cultures with high fish intake have good health outcomes, and that omega-3 supplementation fails to recover those benefits. This makes me feel that eating a lot of fish is likely much healthier than avoiding it, and avoiding it is probably leading to nutrition deficiency, including possibly selenium, which is a much bigger risk than the potential for mercury toxicity.
[1] https://www.sciencedirect.com/science/article/abs/pii/S01618...
There are dangerous caveats to this advice! For example, don't feed infants fish or shellfish more than this allergy test. They contain one of the highest levels of mercury in the entire food web. Best to wait until the they've grown.