Do incumbents in such a system leverage their status, leading to new kinds of abuses? Arguably yes. That said, SVB was a member bank of the Federal Reserve and it didn't help their shareholders when the market decided that they were over-exposed.
Looking at your assertion from the opposite direction, if you don't trust the Federal Reserve, why would you trust a smaller bank to do a better job? Sure, it's in their long-term interest to self-regulate and operate prudently, but history is full of people acting irrationally because of greed, myopia, confirmation bias etc. The fed at least exhibits moderate transparency of structure and operation - risk is distributed across a set of reserve banks, and the membership and balance sheets of the regional reserve banks are open to scrutiny.
And that's what it basically boils down to. You don't want big banks? Eliminate the Fed.
Eliminate the Fed.
Banks eliminate inherent risk by being backstopped by the fed. A simple example of how this works is FDIC insurance. If a bank had all of its depositors holding $250k or less, it could essentially lend out all of that money with very little risk of getting sued by their depositors.