To be extremely pedantic, I think they’re correct at an implementation level: for most chips, as long as they’re receiving power at all from a power supply, they’ll run, unless a reset line is being held high. In most consumer electronics, that reset line is held high by default (by power coming off the wall or from the battery), unless power is supplied to a NAND gate coming from (a flip-flop in front of) the power switch on the front of the computer.

To “turn on” a computer (using the push-button at the front, rather than by flipping a power-supply toggle-switch) is, then, actually to SET that flip-flop, feeding a high input to the NAND gate, which in turn will turn off the reset line to the CPU.

(And, vice-versa, if your computer has a “reset button” on the front like some old computers do, that one throws the flip-flop back to its RESET state, which puts the NAND gate back low, which brings the reset line to the CPU back high. Wiring for push-button toggles is weird!)