Some of the early politicians nd early presidents (jackson) had some odd ideas about central banks and Jackson in particular actually caused considerable financial problems.

Also private banks did not like being regulated by a central bank