News for hackers. Stuff that matters.
Re:Postgres 9.2 will feature linear read scalability up to 6... by justinsb on Tuesday April 03, 2012 @03:42PM (#3794454)
They may be known for their home-grown databases, but they shouldn't be known for their use of them:

Amazon's primary database is Oracle. Dynamo is used for their shopping carts i.e. for storing sessions; Memcache would probably work just as well.

Facebook's database is MySQL, with sharding and Memcache. I was under the impression they stopped using Cassandra entirely?

Google's business (advertising) is built on MySQL, as are many of their sites e.g. YouTube. I think the newer Google-developed sites (e.g. GMail, Reader etc) are indeed built on BigTable.

Re:Postgres 9.2 will feature linear read scalability up to 6... by haberman on Tuesday April 03, 2012 @11:29PM (#3796075)
> Amazon's primary database is Oracle. Dynamo is used for their shopping carts i.e. for storing sessions; Memcache would probably work just as well.

What? Dynamo is a distributed, persistent, highly-available storage system with incremental scalability and advanced techniques for dealing with slow or unavailable servers. Memcache is a single-process daemon that vends an in-memory hash table via TCP. They are not even remotely comparable.

I'm not dissing memcached, it's cool and useful, but its scope is far, far more limited.

In particular, have you noticed that when you add something to your shopping cart on Amazon but don't buy it, it's still there months or years later? The data doesn't disappear just because some process that was holding the data in-memory crashes.

Re:Postgres 9.2 will feature linear read scalability up to 6... by justinsb on Wednesday April 04, 2012 @01:03AM (#3796255)
I can't say I've ever put something in my cart, not logged in to Amazon for months, gone back to see it still there, and thought "wow - they must use some sort of webscale storage solution - I'm now going to buy the thing I didn't want to buy six months ago."

From the point of view of solving the actual business problem, therefore, I think something based around Memcache would work just fine!

Dynamo is very useful for persuading you that working for Amazon would be interesting, though.

Re:Postgres 9.2 will feature linear read scalability up to 6... by haberman on Wednesday April 04, 2012 @02:07AM (#3796394)
Have you ever worked in an environment at the scale of Amazon? Machines go down all the time. I used the "six months" example to illustrate that the shopping cart is persistent, but the machine crash could just as easily happen the moment before you push "checkout." Losing shopping cart data just because one machine crashed is totally unacceptable.

Programming distributed systems that must survive machine failure and network partitions is a completely different ball of wax compared to simple web programming. If you haven't done it before, you would not believe how much more complicated it is.

Here is an extremely simple example. Suppose you're a radio station that's taking phone calls from people and you want to give an award to the 5th caller. Implementing a program that does this for a single machine is easy, and could be accomplished with a program something like this:

  import SocketServer

  class MyHandler(SocketServer.BaseRequestHandler):
    def handle(self):
      self.server.caller += 1
      if self.server.caller == 5:
        self.request.sendall("Congratulations, you are the 5th caller!\n")
      else:
        self.request.sendall("Sorry, you're caller #%d\n" % (self.server.caller))

  server = SocketServer.TCPServer(("localhost", 9999), MyHandler)
  server.caller = 0
  server.serve_forever()
Using this program, you can "call" the program by doing "telnet localhost 9999" and the program will tell you what caller you are. This took me about 5 minutes to write, and I'd never used this Python API before.

Now imagine that you want to implement this same logic, but using a cluster of machines that could go down at any time. You want the group of machines to form "consensus" about which number each caller is; consensus in this context just means that the group of machines arrives at a single answer, and any machine you ask will give you the same answer.

Finding an algorithm that can do this robustly is so difficult that it was a major breakthrough when one was discovered in 1988. It's called Paxos and you can read about it here: http://en.wikipedia.org/wiki/Paxos_(computer_science) Even though it has been known for over 20 years, it is still a complex topic that very few people understand the details of.

The point of all of this is just to say; you can't compare a single-process in-memory cache to a distributed and fault-tolerant system. They are completely different beasts, and many business problems do indeed need the latter.

Re:Postgres 9.2 will feature linear read scalability up to 6... by justinsb on Wednesday April 04, 2012 @04:01PM (#3799553)
I know about Paxos. I know when to use it, and when not to use it.

Seeing as you came up with the example, using Paxos for an "Nth caller wins" is a really bad idea - the Nth caller to your switchboard likely won't be the one selected. You probably _need_ a single server system, like the example you wrote (but without the threading errors.)

Re:Postgres 9.2 will feature linear read scalability up to 6... by haberman on Wednesday April 04, 2012 @06:55PM (#3800307)
Then why would you compare dynamo and memcached? Or suggest that Amazon could use an in-memory cache as the primary store for shopping cart data? It makes no sense.

If you want to be particular about ordering, then you can always use paxos to elect a master that handles everything serially and failsover.

Re:Postgres 9.2 will feature linear read scalability up to 6... by justinsb on Wednesday April 04, 2012 @07:28PM (#3800447)
I see you work at Google. How about advancing the conversation by telling us how Google stores its sessions?
Re:Postgres 9.2 will feature linear read scalability up to 6... by haberman on Wednesday April 04, 2012 @10:37PM (#3801023)
Let's make a deal. I'll start "advancing the conversation" once you stop misleading people (ie. admit that your initial sweeping claims were incorrect).
Re:Postgres 9.2 will feature linear read scalability up to 6... by justinsb on Thursday April 05, 2012 @01:02AM (#3801374)
Deal. My initial sweeping claims were incorrect.

So: How does Google store its sessions?

Re:Postgres 9.2 will feature linear read scalability up to 6... by haberman on Thursday April 05, 2012 @11:16AM (#3802902)
So I'm not an expert in Google's front-ends and I'm not sure that Google stores "sessions" in the way you'd generally think of them in web apps. But usually when we have requirements like what you'd need for sessions (highly available, low-latency, highly scalable) we use Megastore:

http://www.cidrdb.org/cidr2011/Papers/CIDR11_Paper32.pdf

http://www.readwriteweb.com/cloud/2011/02/megastore-googles-...

Megastore is a layer on top of Bigtable that adds indexing, synchronous replication across data centers, and ACID semantics within small partitions called "entity groups."

Re:Postgres 9.2 will feature linear read scalability up to 6... by justinsb on Thursday April 05, 2012 @12:02PM (#3803160)
Good deal! Agree that traditional sessions are best avoided, but good to know that megastore is suitable for session data.
Re:Postgres 9.2 will feature linear read scalability up to 6... by pooriaazimi on Tuesday April 03, 2012 @04:24PM (#3794651)
Thanks. I didn't know about Amazon's... Can you provide more links/sources on this matter?
Re:Postgres 9.2 will feature linear read scalability up to 6... by justinsb on Tuesday April 03, 2012 @04:43PM (#3794733)
Here's a 2007 post: http://highscalability.com/blog/2007/9/18/amazon-architectur...

In short (and filling in some of the gaps with my interpretation), they used "one big Oracle database" up until 2001. The webapps originally talked directly to the database (presumably with local caching), but they introduced an application tier later on. I'd imagine this was as much for data integrity reasons.

As they outgrew the one big DB approach, the service approach also let them split their database along service boundaries. So each service still used one big DB, but there were many services.

My understanding is that now there are hundreds of services, and each service is run by a team that can choose their own internal components. But each team is directly responsible for their uptime; if their service breaks the developers get paged in the middle of the night. Traditional databases are therefore still widely chosen; even if sexy technologies like Dynamo get the press.

"Dynamo for show, Oracle for dough" to mutilate an old golfing expression :-)

Re:Postgres 9.2 will feature linear read scalability up to 6... by pooriaazimi on Tuesday April 03, 2012 @05:02PM (#3794815)
Thank you.