News for hackers. Stuff that matters.
Re:Ask HN: Why does S3 still not support append? by whatnotests on Wednesday December 16, 2015 @06:22PM (#10748174)
All excellent points, especially about cost.

I've considered "faking" the append functionality by making a new file per append action, then performing a periodic compaction.

Even compaction-via-combine-and-delete-old is clunky.

    aws s3 combine --target s3://bucket-name/output-file.txt \
      s3://bucket-name/input-file-1.txt \
      ... \
      s3://bucket-name/input-file-n.txt
I, for one, would pay extra for that.
Re:Ask HN: Why does S3 still not support append? by benjiweber on Thursday December 17, 2015 @01:51AM (#10750016)
The lack of read-after-delete consistency makes this tricky.

https://aws.amazon.com/s3/faqs#What_data_consistency_model_d...

I've seen "eventually" consistent mean up to 24hrs in the face of problems. Several minutes seems common when versioning/bucket replication is enabled.

Re:Ask HN: Why does S3 still not support append? by mikgan on Thursday December 17, 2015 @02:38AM (#10750135)
I can second that. Personally I'd like to see them support symbolic links so version controlling and rolling deployments of static websites becomes a little easier.
Re:Ask HN: Why does S3 still not support append? by maxims on Thursday December 17, 2015 @03:55AM (#10750316)
This is actually pretty kewl. You could potentially branch the current CLI from the GitHub repo and add that functionality in. Ideally the flow would work something like the following:

  1. Start a multipart object upload
  2. Issue "Upload Part - Copy" requests for each part of an object ( http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html )
  3. Complete the multipart object upload
Alternative flow:

  1. Enable bucket versioning
  2. Download the parsed S3 objects
  3. Start a multipart object upload to S3 with the specified target object as the object name
  4. Reupload the parsed S3 objects as parts of a single multipart object upload
  5. Delete the previous parsed objects once the multipart object upload is complete (a delete marker should be added to the top of the version stack, but the previously stored version should still be available if you specify its handle/version id).
Edit: changed formatting