Those middlemen run the federal reserve. Good luck trying to remove banks from power without migrating to a different currency altogether.