Again, right now 1 in 100 photons will interact with an optical switch. This effect multiplies with each level of transistors. This means you have to "run" your circuit potentially something like 10^100 times to get all of the transistors in your path to work on the same photon.

That's pretty rough.