Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

Re:Overfitted a 900KB Transformer to Compress a 100MB CSV in... by tae0086 on Wednesday June 24, 2026 @10:11PM (#48667986)

Neat approach. Since the 900KB model ships with the compressed file, is there a file size below which the model overhead just eats the gains? Curious where the crossover is.

[ Parent | on HN ]

Re:Overfitted a 900KB Transformer to Compress a 100MB CSV in... by spidy__ on Thursday June 25, 2026 @04:18AM (#48670504)

For the model overhead to become significant enough to eat into the gains, the file size would need to be fairly small, right? I assumed nobody would use this for compressing anything below 100 MB.

I tested with 100 MB files because anything larger takes a long time to evaluate. The actual target was at least 1 GB, and in that case I would use a 100 MB model (Shannon entropy rules).

I also tried it on a 100 MB Photoshop file and was able to compress it down to 45 MB, whereas ZIP could only get it down to 60 MB. So yeah still not losing gains.

[ Parent | on HN ]