> Branches and tags are just pointers to a commit

Technically that's correct, but it is not how branches are thought of and used. Conceptually a branch is, well, a line of development, a sequence of commits with a name. (Actually it is not necessary just a sequence, but this is not important here)

I think the implementation of branches in Git as just a kind of fancy tags is one of the sources of confusion about/around Git.

And I love Git, don't get me wrong! But some things in it make things harder than necessary.