Is there any reason why you are not streaming data directly from source into Elasticsearch by running Logstash at source servers?

Our client has been storing months of tracking events on S3 via Kinesis. This seems to be a simple, cheap and reliable solution to store billions of datapoints. We can then import that data into ElasticSearch, Amazon Redshift, etc.

ive thought about this too when considering an architecture like the one described ..seems as though storing in S3 is a good first stop for the data as it takes very little care and feeding for a massive amount of data.

then, depending on your query and data processing needs you can look to other tools such as elastic search, or elastic map reduce or xyz db.

see also : http://petewarden.com/2010/10/01/how-i-ended-up-using-s3-as-...

S3 is quite limited when it comes to listing a subset of objects as scope using prefixes. So can fetch all objects starting with "2014/10/10" but you can't fetch the last 30 created objects for instance.

I get the impression that this set up was for a client and a bit of an R&D project. I guess his client just made their logs available on S3 for him to consume.