I feel there is a useful analogy in what happens in toxic environments. In areas where there are high levels of arsenic, for example, you find a limited ecology of organisms adapted to tolerate it, and I think I am right to say that they do not do well outside of this environment, as the mechanisms of tolerance are sub-optimal where they are not needed.

The most relevant situation would be where the environmental toxin is mutagenic. My uninformed guess is that adaptation to that environment would typically involve mechanisms to reduce susceptibility to the toxin's mutagenic effect, and, as in the case with other toxins, organisms so adapted would be out-competed in areas where the toxin is not present.

I guess we have examples in the microbes which have adapted to live in areas of high radiation, but I do not know how they fare elsewhere.

Update: D. radiodurans is an example, but it has been suggested that its tolerance is simply a side effect of a mechanism for dealing with prolonged cellular desiccation - another sort of environment where they do well.

https://en.wikipedia.org/wiki/Deinococcus_radiodurans#Evolut...

There were no organisms selecting to reproduce either. Of course, natural selection is largely a random process that organisms are subject to rather than a process that requires them to be active agents.

That's cause and effect reversed! Organisms don't select, it's selection pressure that determines which organisms get to the age of reproduction and whose offspring are viable. Some outside factor (environment, another organism) needs to supply the pressure. Absent pressure life will occupy whatever room you give it until it runs up against some kind of boundary and then that pressure will take over again.