News for hackers. Stuff that matters.
Re:Stable Diffusion 1.5 by liuliu on Friday October 21, 2022 @07:48AM (#33287010)
Model can be retrained (with some money). But data is harder. I cannot backup LAION 5B unfortunately. If you can, please do! (About 200T)
Re:Stable Diffusion 1.5 by pmoriarty on Friday October 21, 2022 @08:31AM (#33287319)
A distributed backup might be possible.

Get 200 interested people backing up 1 TB each and you have your 200 TB backup.

With redundancy and error correction data added to the mix, you should be able to lose a certain percentage of participants and still have access to the full, error-free backup.

Re:Stable Diffusion 1.5 by liuliu on Friday October 21, 2022 @09:52AM (#33287992)
Yeah, need to write the program and distributed in r/datahoarder

This should be next on my list since my current project depends on SD model and having data backed up gives me confidence that I can get rid of all their stuff if needed.

Re:Stable Diffusion 1.5 by MayeulC on Friday October 21, 2022 @11:25AM (#33289007)
Hmm, you can just create a torrent out of it. Either as a single file (impractical, but you can just avoid downloading it fully), or chunk it into multiple files.

You don't even need to store it all at once on your computer: stream it and generate checksums on the fly. Then distribute the torrent, and seed sections at a time. It can also be distributed on IPFS.

I've seen a lot of torrents being used for distributing neural network (mostly stable diffusion forks).

Re:Stable Diffusion 1.5 by liuliu on Friday October 21, 2022 @11:32AM (#33289082)
Good point. Let me think through this. I have about 15T free space so this can be seeded separately in ~12 batches.
Re:Stable Diffusion 1.5 by MayeulC on Friday October 21, 2022 @07:24PM (#33294018)
You can also distribute different torrents, it's easier.

Or multiple IPFS CIDs. I think you can have a "directory" (CID) that contains multiple CIDs, and only need the content hashes to build it.

You can also publish multiple CIDs and ask people to seed random ones; that's how Libgen does it (and is similar to the multiple torrents concept).

The same file can be used to seed both torrents and IPFS.

Re:Stable Diffusion 1.5 by l33tman on Friday October 21, 2022 @04:33PM (#33292724)
Why would you, though? It's just a list of 5B URLs. Some might go down, some new might go up. But it's not like any government body can suddenly take down all photos on the whole internet...