News for hackers. Stuff that matters.
Re:How Tarsnap uses Amazon Web Services by bentoner on Sunday December 14, 2008 @03:54AM (#397299)
Great post! I don't understand the paragraph about the cost of the PUTs and GETs though. The saving you get by batching writes seems marginal. Can you give some numbers?
Re:How Tarsnap uses Amazon Web Services by cperciva on Sunday December 14, 2008 @10:29AM (#397392)
The average block size the tarsnap server sees is about 30 kB (the tarsnap client tries to produce blocks of 64 kB on average, but then it compresses them individually before sending them to the server). This means that for every GB uploaded, there are about 33 thousand blocks.

S3 PUTs cost $0.01 per thousand PUTs, so writing each of the blocks as an individual S3 object would cost $0.33 / GB for PUTs (plus the normal $0.10 / GB for bandwidth).

Re:How Tarsnap uses Amazon Web Services by jrockway on Sunday December 14, 2008 @12:34PM (#397456)
Just out of curiosity, why such small blocks?
Re:How Tarsnap uses Amazon Web Services by cperciva on Sunday December 14, 2008 @12:52PM (#397480)
There are several advantages and disadvantages to different block sizes; but most significantly, larger blocks would make tarsnap less efficient at identifying duplicate data in the (very common) case where part of a file is modified. In the end it came down to weighing all the factors and picking a value which worked well.
Re:How Tarsnap uses Amazon Web Services by andr on Sunday December 14, 2008 @08:03AM (#397345)
How exactly do you batch writes? Do you save 2 blobs in 1 file? Or some kind of keep-alive that Amazon counts as a single request?
Re:How Tarsnap uses Amazon Web Services by cperciva on Sunday December 14, 2008 @10:31AM (#397393)
I aggregate blobs together (up to 4 MB at once) and store them to S3 as a single S3 object. The tarsnap server keeps track of which part of the object corresponds to which blob.