News for hackers. Stuff that matters.
Re:A Tour of Amazon's DynamoDB by jedschmidt on Monday January 30, 2012 @11:29AM (#3529367)
I've been waiting for Mathias to write something like this, and I'm glad he didn't waste any time diving in. For those who don't know him, he has more breadth in the area of emerging datastores than anyone I know; he literally wrote the book on Riak[1].

I've also been acquainting myself with the DynamoDB API over the past week, and am building a node.js binding that I hope will abstract away most of the esoteric aspects of interacting with it. It has full API coverage currently and is tested in Travis[3], so now I'm writing the high-level interface. So far I've covered about half of the operations DynamoDB offers, but would love to hear any ideas/feedback.

    [1] http://riakhandbook.com/
    [2] https://github.com/jed/dynamo
    [3] http://travis-ci.org/jed/dynamo
Re:A Tour of Amazon's DynamoDB by ryanfitz on Monday January 30, 2012 @03:07PM (#3530430)
Looks like we are building a very similar node.js client for dynamo. https://github.com/Wantworthy/dynode
Re:A Tour of Amazon's DynamoDB by jbellis on Monday January 30, 2012 @11:54AM (#3529476)
I'm still struck by what a strong endorsement DynamoDB is of the design decisions we've been making in Cassandra over the last couple years. Composite keys, distributed counters, ...

More details: http://www.datastax.com/dev/blog/amazon-dynamodb

Re:A Tour of Amazon's DynamoDB by leef on Monday January 30, 2012 @02:55PM (#3530368)
Dynamo as described in the original paper was not made into a web service.

Facebook took up the Dynamo torch and created Cassandra... then switched to HBase after trying to use Cassandra in production.

Have you ever stopped to wonder about what Amazon might have taken OUT of Dynamo before launching DynamoDB? Things that Cassandra still has and just might be holding it back?

That to me seems much more important than just saying "DynamoDB has feature X that Cassandra has had for years! We're so smart!"

Re:A Tour of Amazon's DynamoDB by jbellis on Monday January 30, 2012 @04:04PM (#3530654)
There's less to write about there... One of the first Cassandra decisions was dropping vector clocks, reasoning that vector clocks weren't worth the additional complexity for 99% of uses (once you move from key/value to rows + columns). DynamoDB also switched to items/fields and dropped vector clocks (or at least does not expose them).

The original Dynamo was plain key/value with O(1) routing and vector clocks; there's not much else to strip out. :)

P.S. Facebook built Messages on HBase instead of Cassandra for political reasons rather than technical, and shards HBase to mitigate the availability problems it has otherwise. Facebook never ran an Apache Cassandra release in production.

Re:A Tour of Amazon's DynamoDB by foobarbazetc on Tuesday January 31, 2012 @12:38AM (#3532027)
I'm struck by how you somehow manage to pimp both Cassandra and DataStax on every post which has something to do with databases and/or datastores. :)
Re:A Tour of Amazon's DynamoDB by j2labs on Monday January 30, 2012 @12:37PM (#3529685)
Didn't you folks design Cassandra after the Dynamo paper though?
Re:A Tour of Amazon's DynamoDB by thamer on Monday January 30, 2012 @01:07PM (#3529853)
Composite keys (multi-dimensional) and distributed counters were not in the Dynamo paper. They were added to Cassandra, which is indeed based partially on Dynamo.
Re:A Tour of Amazon's DynamoDB by amix on Monday January 30, 2012 @01:40PM (#3530005)
Cassandra initial team was lead by Avinash Lakshman who was one of the authors of Amazon's Dynamo. So it's not strange that Cassandra is hugely inspired by Dynamo (and vice versa).
Re:A Tour of Amazon's DynamoDB by stonemetal on Monday January 30, 2012 @01:10PM (#3529869)
Yes, but the Dynamo paper offers two operations Get(key) and Set(key, value). Anything Cassandra adds atop that they did themselves i.e. column families, secondary indices.
Re:A Tour of Amazon's DynamoDB by chargrilled on Monday January 30, 2012 @01:46PM (#3530038)
How does Cassandra stack up against HBase these days?

Both projects seem to be moving so quickly that it's really hard to find an up to date comparison.

Re:A Tour of Amazon's DynamoDB by cleaver on Monday January 30, 2012 @01:43PM (#3530018)
DynamoDB is not open source, at least according to a blog comment by Jeff Barr of AWS. (http://aws.typepad.com/aws/2012/01/amazon-dynamodb-internet-...)

I'm certainly not going to commit to platform lock-in like that. I know a lot of folks were hit hard by Googles app engine price changes and it could happen again here with DynamoDB.

Re:A Tour of Amazon's DynamoDB by techscruggs on Monday January 30, 2012 @01:53PM (#3530064)
Then you don't use S3?
Re:A Tour of Amazon's DynamoDB by cleaver on Monday January 30, 2012 @02:01PM (#3530103)
I don't actually, but I feel there is less lock-in. It wouldn't be hard to move off.

Also, the risk of S3 pricing changing greatly is constrained by the wide adoption. If DynamoDB failed to become pervasive, it would be relatively easy for Amazon to increase the price to make an underutilized service profitable. OTOH, if DynamoDB does become as popular as S3 your exposure to risk will be much less.

Re:A Tour of Amazon's DynamoDB by spullara on Monday January 30, 2012 @02:17PM (#3530181)
It would be pretty easy to implement the DynamoDB API on top of HBase so I wouldn't worry too much about that.
Re:A Tour of Amazon's DynamoDB by sciurus on Monday January 30, 2012 @01:56PM (#3530087)
There's an open implementation of S3.

http://open.eucalyptus.com/wiki/EucalyptusStorage_v1.4

Re:A Tour of Amazon's DynamoDB by siculars on Monday January 30, 2012 @04:26PM (#3530709)
I would second Mathias' thoughts in that this does feel more bigtable/column oriented than traditional dynamo. Dynamo and its derivative, Riak (not from amazon), make no attempt to determine data types or schema in any way. The fact that dynamodb can do range scans and has various counting features leads me to believe that it is more hybridized along the lines of Cassandra. Either way, it is a welcome addition to the nosql toolset that developers have to chose from today. I will certainly look at it more closely.
Re:A Tour of Amazon's DynamoDB by rbranson on Monday January 30, 2012 @04:30PM (#3530725)
Actually Riak can be used in a way in which it is content-aware of JSON and XML documents.

http://wiki.basho.com/Riak-Search---Indexing-and-Querying-Ri...

Re:A Tour of Amazon's DynamoDB by siculars on Monday January 30, 2012 @04:41PM (#3530756)
Search and Indexing were addons to the initial Riak product that were born from user demand. Only recently (1.0 I believe) was Search integrated into the main Riak distribution. Riak has no capability of updating values in place, ie. a partial update and no means of ordering or pagination. Some of these features are available via mapreduce queries or Search/Index as you noted.

I'm by no means harping on Riak. I actually use it on a number of projects. But reading about DynamoDB's capabilities does not conjure Riak, it conjures Cassandra and HBase.

Re:A Tour of Amazon's DynamoDB by rbranson on Monday January 30, 2012 @04:45PM (#3530771)
I agree with you on other counts, but Riak is a far cry from the spartan implementation of Dynamo outlined in the paper.
Re:A Tour of Amazon's DynamoDB by taylorbuley on Monday January 30, 2012 @12:37PM (#3529686)
Does anyone know in which language DynamoDB is implemented? I've read somewhere that SimpleDB is done in Erlang. Is that the case with DynamoDB as well? I've been reading about Ets and Dets in Erlang and it makes me wonder whether they have anything to do with either of these data stores.
Re:A Tour of Amazon's DynamoDB by gigq on Monday January 30, 2012 @01:53PM (#3530062)
I haven't seen it written anywhere, but judging by the __type in errors like this, I would say Java:

  {"__type":"com.amazon.coral.validate#ValidationException",
  "message":"One or more parameter values were invalid:
  The provided key size does not match with that of the schema"}
Re:A Tour of Amazon's DynamoDB by antirez on Monday January 30, 2012 @02:50PM (#3530350)
About pricing, it is interesting how caching is apparently not an effective option to reduce the bill in a significant way since data size is a big component of price.
Re:A Tour of Amazon's DynamoDB by justincormack on Monday January 30, 2012 @04:09PM (#3530673)
Thats because it is SSD based, so size starts to matter again.
Re:A Tour of Amazon's DynamoDB by antirez on Monday January 30, 2012 @04:37PM (#3530748)
Yes but real hardware with an SSD disk plugged inside could deliver excellent performances with many reasonable on-disk databases, with a fraction of the cost per megabyte, with many many millions of keys.

It is like that other than the fact you don't have to manage the DB, that is a good point, here the best reason for many small-mid business to use such a service is since they are already on EC2. May work in the short term but I see something odd about this model.

Re:A Tour of Amazon's DynamoDB by SaltwaterC on Monday January 30, 2012 @10:59AM (#3529235)
The API doesn't bother me as much as the temporary authentication via STS. Temporary credentials for database access? Seriously? I am the only one who sees how ridiculous this is?
Re:A Tour of Amazon's DynamoDB by amock on Monday January 30, 2012 @06:59PM (#3531124)
What's wrong with using temporary credentials?
Re:A Tour of Amazon's DynamoDB by SaltwaterC on Sunday February 12, 2012 @10:50AM (#3582070)
They add useless latency and another point of failure. And sometimes, the AWS APIs DO fail.
Re:A Tour of Amazon's DynamoDB by amock on Sunday February 12, 2012 @04:20PM (#3583057)
Temporary credentials are used specifically to reduce latency, since you get a token that is valid for a period of time and doesn't need to be checked against the auth service on every call. Since the credentials last for 12 hours the work to retrieve them should be negligible. Because they don't need to be checked against the auth service on every call it seems like they would more resilient to API failure than standard AWS credentials.
Re:A Tour of Amazon's DynamoDB by SaltwaterC on Tuesday February 14, 2012 @05:31AM (#3589445)
You still need to provide an access key id and secret access key with that session token as returned by Security Token Service. And properly sign every request with those credentials. With a proper session token, but invalid credentials, the request fails with HTTP 400: "__type: com.amazon.coral.service#InvalidSignatureException, message: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.". Amazon still checks the signature. The same logic that could be used for IAM credentials that don't expire, without the need to check for a valid session token. If it is an Amazon screw up, that static credentials take more time to evaluate vs temporary credentials - that's not out fault. It is a pure Amazon screw up. The signing logic is still there for every API call. So (apparently?) we're back to square one: extra requests and wrapper logic for zero benefits, and worse overall experience. The session credentials aren't stateful. I specifically checked for this behavior. Therefore what you say, doesn't seem to happen in reality.

And for the love of God, don't blindly trust the documentation / what the AWS folks say. As an AWS library author myself, I had a lot of fun debugging failed requests because the smarty who wrote the signing procedure docs forgot to mention some HTTP headers that are mandatory to sign. I had to reverse engineer an official SDK in order to patch my own code. While being implemented as the docs say, the signing method failed at every request, although I had a valid session token. Trial and error is always a broken way of developing things due to broken docs. If you're an AWS employee, please send my regards to the documentation folks.