News for hackers. Stuff that matters.
Re:How Tarsnap uses Amazon Web Services by cperciva on Sunday December 14, 2008 @10:29AM (#397392)
The average block size the tarsnap server sees is about 30 kB (the tarsnap client tries to produce blocks of 64 kB on average, but then it compresses them individually before sending them to the server). This means that for every GB uploaded, there are about 33 thousand blocks.

S3 PUTs cost $0.01 per thousand PUTs, so writing each of the blocks as an individual S3 object would cost $0.33 / GB for PUTs (plus the normal $0.10 / GB for bandwidth).

Re:How Tarsnap uses Amazon Web Services by jrockway on Sunday December 14, 2008 @12:34PM (#397456)
Just out of curiosity, why such small blocks?
Re:How Tarsnap uses Amazon Web Services by cperciva on Sunday December 14, 2008 @12:52PM (#397480)
There are several advantages and disadvantages to different block sizes; but most significantly, larger blocks would make tarsnap less efficient at identifying duplicate data in the (very common) case where part of a file is modified. In the end it came down to weighing all the factors and picking a value which worked well.