The logic around only accepting a block if you previously saw a signed hash is vulnerable to sybil attacks. An attacker could run thousands of nodes on the p2p network, never relay anyone else's signed hashes, steal the proof-of-works from completed blocks and then broadcast signed hashes of its own blocks with the stolen pows and then broadcast those blocks. Many users might find themselves connected to only nodes run by the attacker and would get tricked into accepting the attacker's chain.
The proof-of-work not depending on the blockchain state means that a miner could queue up lots of complete proof-of-works and then mint a ton of blocks with them all at once to do a 51% attack.
Any solution to this is either going to be centralized or look like proof-of-stake, and at that point, the proof-of-work is extra and not actually load-bearing at all.