I don't think so. Basically the idea is just to use whatever data structures you were already using (lists, arrays, hash tables, etc.) and just keep them around. Since you already have this in your code, you avoid doing anything with a DB. There are tools in a variety of languages and frameworks to help with the persistence aspects.

I don't see how that could take more time than doing both the normal program data structures and the DB.

On further thought I think you're right. The need for joins can be alleviated with pointers, and sorting and grouping can be done in the application.

The tricky part would be the synchronization and transactional aspects, and if the system became sufficiently distributed that part could get fairly nasty.

You're still missing the point. Joins and sorting and grouping in SQL are what you use to populate the real data structures. Whether you know it or not, in all but the most simplistic CRUD application, you've gotta write code to deal with all of those data structures. The database is always in addition to that complexity. The argument being made is that you can leave out the database entirely in some environments, and not add that complexity.

The entirety of this thread is questioning the pervasive use of databases in web applications. There are some where it makes excellent sense (accounting, CRM, ERP, the stuff that's been the strong-hold of Oracle and SAS for years). And others (wikis, blogs, forums, photo galleries, etc.) where it may not make good sense, because you introduce hundreds of unnecessary operations and significant additional database support code. That's not to say a database is never the right solution for these problems (when your problem starts looking relational, you should start looking at relational databases, because your Ruby or Python or Lisp implementation is going to be worse than what PostgreSQL or MySQL have). And, as someone else mentioned (but got modded down)...when you start building your own flaky transactional layer, then it may be time to consider a database that has good transactions support.

Here's an actual example, instead of all this pontificating. It's not that impressive but it illustrates the point.

I had to write an external "chat history" module (i.e., always send the last N lines of a conversation to a client as soon as client joins) for an IRC server, in python. (it's a long story)

Your standard RDBMS solution would probably involve the ircd logging directly or indirectly to a database, with a python script handling requests and doing selects out of that db. On a network with thousands of users, this will load a db machine down with IO operations. This also has to happen quickly, so that new text is added to the history log in near real-time. Batching is probably out of the question.

My solution was to have the ircd pass off its strings to another program (in this case, I used python) via a named pipe (this could also have been done via a socket, to aggregate multiple servers or do other neat tricks), and then have the logger app load these into memory - in this case, a giant python dictionary keyed on channel name which pointed to a constantly-updated ring buffer class for each channel. This ring buffer held the last N lines of conversation.

Writes were simple - h[c].write('text')

Lookups were simple - h[c].get()

Fast writes, fast reads, and disk i/o was batched for efficient logging to disk, at which point you deal with the data in batches and maybe store it into a db if you want to do something more advanced like with SQL.

Of course this could all be done with C within the ircd itself as well, using a hash table and ring buffers as well.

It was actually much faster (and logically simpler) than writing a DB solution, and I'm certainly no master hacker.

Funny - I'm exactly the opposite. Making something in memory can be as simple as creating a bunch of objects and stashing them in lists, vectors, hash-tables, etc. Retrieval is just traversing those lists, vectors or hash-tables. Whenever I use sql I find myself guessing at the correct almost-human syntax and getting it wrong. Keeping it all in lisp means I don't have to switch languages constantly while developing.

One of my spare-time projects, lispdoc.com, uses an in-memory approach (although to be fair it is only indexing a few hundred MB of static data, so it's no effort to keep it all in memory on a single box).