I rechecked some random internet articles, and the brain contains about 10^15 synapses. They operate at about 10Hz (~10 PFLOPS). If we arbitrarily assume that each synapse operation is equivalent to 1 floating point operation on a computer, then this is about 3 orders of magnitude more than a good GPU (~10 TFLOPS). Which is actually a lot less than I thought. Still, the brain's basic structures have been optimized over ridiculous amounts of time, the training data those get while you exist is years of high-resolution images/sound/etc, and being a smaller in-skull device it has advantages of lower latency than, say, a cluster used to train ML models.

I think the link between 1 floating point operation and 1 synapse is probably a bit dubious.