Re:How Tarsnap uses Amazon Web Services
by bentoner on Sunday December 14, 2008 @03:54AM
(#397299)
Great post! I don't understand the paragraph about the cost of the PUTs and GETs though. The saving you get by batching writes seems marginal. Can you give some numbers?
[ Parent
| on HN ]
The average block size the tarsnap server sees is about 30 kB (the tarsnap client tries to produce blocks of 64 kB on average, but then it compresses them individually before sending them to the server). This means that for every GB uploaded, there are about 33 thousand blocks.
S3 PUTs cost $0.01 per thousand PUTs, so writing each of the blocks as an individual S3 object would cost $0.33 / GB for PUTs (plus the normal $0.10 / GB for bandwidth).
Just out of curiosity, why such small blocks?
There are several advantages and disadvantages to different block sizes; but most significantly, larger blocks would make tarsnap less efficient at identifying duplicate data in the (very common) case where part of a file is modified. In the end it came down to weighing all the factors and picking a value which worked well.
How exactly do you batch writes? Do you save 2 blobs in 1 file? Or some kind of keep-alive that Amazon counts as a single request?