Just as with Bitcoin, you have to know the genesis block.
Which is generally hardcoded in the client you run. So in that sense you trust the binaries you download, or the source code that you compile. The advantage of PoW consensus is that it's much easier to tell the real chain from a fake one, as the latter will not have much more work performed on it.
Even with the genesis block fixed, control of the initial keys embedded in it allows you to generate arbitrary alternative blockchain histories that without PoW are difficult to distinguish from the "real" one.
With a PoW blockchain you do not need to trust the genesis block. You can calculate the total amount of work represented by any blockchain which provides an independent measure of trust. Total proof of work can be calculated with just a copy of the blockchain data without trusting the software client. Nothing like that exists with proof-of-stake.
If the assumption is that downloaded binaries must be trusted, one could simply use a key embedded inside that binary to verify the identity of a centralized database. No need for proof-of-stake.
Some other group could follow this exact protocol and also call their blockchain Algorand.