News for hackers. Stuff that matters.
Re:Internet Archive as a default host-of-record for startups by AlexanderTheGr8 on Tuesday December 21, 2021 @04:20PM (#29642251)
What compression does IA use to store websites? Using a 2x better compression will allow them to store 2x more websites/content.

I am doing some compression research, and would love to help IA in any way I can. There are some amazing SOTA compression algorithms available now.

And if IA ignores images/video, and focuses only on text, they can store an insane amount of websites at a very low cost.

Re:Internet Archive as a default host-of-record for startups by jrwr on Tuesday December 21, 2021 @10:17PM (#29645222)
right now, most of it is coming in as ZSTD with a central index. older stuff was gzip
Re:Internet Archive as a default host-of-record for startups by AlexanderTheGr8 on Saturday December 25, 2021 @11:45AM (#29684652)
I am working on a ML-based compression which will be slow to compress/decompress but will give much better compression than zstd/gzip (even 2-3x better). Do you think that it's a useful algorithm for archival reasons or for storing huge amounts of data in cloud?