You still have to trust who's getting money into and out of the system. And they're usually related to banks and legal systems.

Most fraud doesn't happen "behind the counter of the bank", but before the money gets there.