Don't start from scratch, take a look at yacy (https://yacy.net/) that already does most of what is discussed here
I would think everyone would run their own “crawler” but maybe you could use a ledger and delegate sites to different workers. If you’re whitelisting sites you could maybe only crawl a link or two deep.
It’d take a lot of cycles while small but if you get a network growing you could even have sub-networks with their own whitelist additions (and every user has a blacklist.)
I'm thinking about playing around with this in my spare time, but that part seems the hardest to do.