maybe very slowly with the just a few workers wearing super expensive hazmat space-suits with their own self-contained breathing systems. that probably wouldn't scale to production volumes but could make sense if you're building a one-off robotic factory.

but nobody answered the question - are the normal outputs of a factory also toxic, or just what goes on inside? would a robotic factory even "solve" the issue?

You can't grind up a circuit board or chip and eat it (or drink it) - which down the line leads to problems:

https://en.m.wikipedia.org/wiki/Electronic_waste_in_Guiyu

That is to say, one would need proper recycling in order to avoid toxicity for the entire life cycle of products.