When we would perform a rolling reinstall of the entire worker cluster (~5500 1U pizza box servers), we would use a custom installer that would utilize Bittorrent to retrieve the necessary RPMs (Scientific Linux) instead of HTTP; the more workers reinstalling at once, the faster each worker would reinstall (I hand wave away the complexities of job management for this discussion).

I'm not super familiar with IPFS (I've only played with it a bit to see if I could use it to backup the Internet Archive in a distributed manner), but I'm fairly confident based on my limited trials that yes, you could build a Debian installer CD to fetch the required packages from an IPFS mirror. No need to even have the file index locally. You simply need a known source of the file index to retrieve, and the ability to retrieve it securely.