"One of us in each world" is shorthand for "one of us in the entire system described by the wave function, with different amplitudes in different states".

There is only 1 detector, and it, along with the detected particle, is in a superposition of states.

Again, this is where the language of "worlds" becomes misleading, and why I don't like it.