Similarly I can definitely see how there would be a mechanistic explanation for intelligence (basically just build a machine that acts intelligently, perhaps even claiming that it's conscious), but there's a distinction between appearing conscious and actually feeling something.

That's the "hard problem" of conciousness stated simply, and the answer to it would seemingly have to involve some quite direct access to some part of brain state physics - meaning that the physical properties involved are quite "basic" in some sense.

As a software developer, this should be simple. Programs have their own state that you can understand without knowing the exact formulations of electrons in the processor that ultimately make up that state.

X = 5 sure but good luck figuring that out looking at the motherboard of a running computer.

In every external way they appear conscious, yet they are not. They are not aware of their own existence.

So consciousness is clearly more complicated than just "summation of changes that happened in our brain", there is something extra there, that we don't understand.

It seems most people want to make consciousness out to be more complicated in some way.