Robots help. The "lights-out" fab has been an industry goal for thirty years at least. (Lights-out = no human operators. I.e. you could turn out the lights and nobody would mind.) We've moved a long way in that direction, but we're not going to remove humans in the forseeable future. The reason is something your question, like many discussions about automation, completely overlooks.

Robots break. All the time.

Robots need recovery when they break. They also need preventative maintenance. Even a fully automated fab has an army of technicians in there turning wrenches 24/7. In fact, the more automated the fab, the more people you need turning wrenches. So you can't have a factory that's a toxic swamp inside so long as it's clean on both ends, because there are people in there. You really don't want to ask them to wear SCBA gear all day long either. Nobody likes that. It's hot, it's uncomfortable, and it doesn't work well in tight spaces.

I get the point of the phrase ("light-out fabrication") but I think it's important to note that robots smart/capable enough to replace humans would probably use visible light in a manner quite similar to the way it is used by humans.

And what's worse is those wrench-turning techs have a different mindset from the test-tube shuffling techs they replace. So when they come in, they're more likely to get exposed to something nasty.

Also, if you're unhindered by things like preventing tumors by isolating chemicals from your squishy meat employees, imagine the kind of Superfund sites you'd get when these places were finally decommissioned.

How do you bootstrap a robot factory if you need the factory's output to make the robots?

maybe very slowly with the just a few workers wearing super expensive hazmat space-suits with their own self-contained breathing systems. that probably wouldn't scale to production volumes but could make sense if you're building a one-off robotic factory.

but nobody answered the question - are the normal outputs of a factory also toxic, or just what goes on inside? would a robotic factory even "solve" the issue?

You can't grind up a circuit board or chip and eat it (or drink it) - which down the line leads to problems:

https://en.m.wikipedia.org/wiki/Electronic_waste_in_Guiyu

That is to say, one would need proper recycling in order to avoid toxicity for the entire life cycle of products.

You'd have a similar situation as that of a nuclear power plant.

Well the "non-waste" output of a nuclear plant (what it actually produces) is just electricity, but are the chips these plants produce themselves toxic following production, or is only the production process toxic? I don't know about toxicity of these things.

Radiation is hard on electronics. Presumably some chemicals are too but it would seem easier to block a chemical contacting a part than it is to keep radiation out.