It's a different sort of 'passive'. There is this thing called https://en.wikipedia.org/wiki/Default_mode_network which 'lights up' in very different ways, depending on what you do, and how. One could argue that is to be expected, because different regions of the brain are exercised. But that's not all of it. Regarding exercise, think of 'Use it, or lose it'. It's mostly the imagination which is exercised while reading. If it isn't, it shrivels.