The dataset is stored in memory on a server written in Arc, in turn running on top of MzScheme. (The source comes bundled with Arc.) It occasionally writes changes to disk, and lazily reloads them when restarted.

The data set is pretty small, though - the id for this comment is just under 700,000, for instance. While it's nice to see that one server running an interpreter can hold up to so much traffic (and this is much more traffic than many people will ever get!), it's not really representative of the cases where worrying about scaling actually becomes necessary.

While it is cool, I wonder if that really was less work than simply storing things in a db. Presumably with a good ORM mapper, caching the data in memory transparently should only be a small configuration setting.

It just seems to me there is so much to worry about when working with the FS directly. Like what happens when the app crashes in the middle of a write? Maybe it is not an issue anymore with modern, journaling file systems, but there are probably some issues left?

Actually, it wouldn't even need an ORM, it's just a graph built out of sexps.

I agree that a real database would still be useful, though at that size it would just be for ACID transactions. OTOH, writing everything using native data structures makes prototyping a lot easier. Lisp sexps, Lua tables, Python dicts, etc. are fantastic for prototyping.

> While it is cool, I wonder if that really was less work than simply storing things in a db.

Yes, it's less work. The introduction of a databases nearly always more than doubles the complexity of a program and the amount of code written. There's a reason developers like in memory prototypes and persistent hash tables, they require less code than any other solution and don't force you to tear apart your model and try and stuff it into tables and put it back together again on every page view.

Not if you factor in the amount of code necessary for ACID transactions, etc. If you make an informed decision that you don't need that level of consistency, sure, but it's probably reasonable to be overly cautious with data by default.

(If you're going to the database and back for every page view, you could probably be using caching more effectively, e.g. memcached.)

ACID transactions have nothing to do with RDBMS, non relational db's can be and are ACID compliant as well. Look at any object database, you'll generally find MVCC and ACID compliance.

The issue is whether your data fits well into tables and rows, or not, and whether it's worth cramming it into tables and rows and continually reconstructing it to get the benefits relational db's generally provide over the alternatives... which are enforced data integrity, language neutrality, and a generic fixed structure capable of allowing flexible querying. To get those, you trade speed and ease of development.

I think we fundamentally agree, but we've been a bit loose with terminology along the way, and have been arguing with misreadings of each others' points.

Have a nice weekend.

Software: Arc

http://arclanguage.org/

Hardware:

http://news.ycombinator.com/item?id=516122

afaik, everything is stored in memory. I might be wrong though.