Anyhow, read the permacoin paper. It’s pretty cool, and it needs a large petabyte data seed to secure the network. Seems like a win-win to me.
The central server has access to the entire file, and hence can compute the hash of any arbitrary chunk. Challenges/verifications don't have to happen all that often ([1] indicates they are looking at once a month) so creating a unique challenge for each user shouldn't be too compute intensive.
For each known user:
- Central server chooses random chunk of each file it wishes to verify the user still has. This could be any length from as long as the hash function to the whole file size and could be offset any number of bits.
- Client is asked to provide the hash of the chunk using the file stored locally.
Precomputing hashes for all possible chunk permutations would take up substantially more space than simply storing the file in the first place. A bad actor would need to store the hash for all the possible chunk lengths starting from every possible start location in the file which is in the order of O(n^2) where n is the stored file size (500GB in this case). For reference that would be about one third of the entire 20PB archive for a single 500GB chunk if using a 256 bit hash function.
1) Someone keeps downloading the full archive and throwing away;
2) Someone wants a file erased, keeping it with the intention of denying access in some future.
--
1) Bandwidth has costs on both sides; just balancing upload among receivers would probably suffice for this not to be a problem.
2) Assigning large random chunks to downloaders should prevent this "chosen-block attack"; add in some global redundancy for good measure and that's probably enough (although I still wouldn't trust this 100% as a primary storage, only as an insurance storage).
This is not a new problem, and most of the obvious solutions work just fine.
Permacoin is a cryptocurrency that uses proof of storage instead of bitocoin’s proof of work:
https://www.cs.umd.edu/~elaine/docs/permacoin.pdf