Database for searching for by the item (news story, game, journal entry), pickle or sexprs for the individual comment threads. The database helps you manage the hopefully large number of news stories you'll get. The tree takes care of individual comments for a single news story.

Many filesystems have trouble with large numbers of files in a single directory, as you'd get with eg. pickle tree to a file per item.

Is this what you are actually doing? How do you store the pickle output?

I'm not - I'm using the approach here: http://news.ycombinator.com/item?id=33888 . That gives me much more flexibility in picking out individual comments from a thread, returning all comments by a specific user, getting all comments posted today, etc. I'm just recognizing this as another possible way to solve the problem, if you don't care about the extra functionality..

As for storing the pickled output:

  dbh.execute('UPDATE articles SET comments = $comments WHERE news_id = $id', 
              comments=pickle.dumps(article.comments))

The comments field can be a normal TEXT field, since the default format for pickle is ASCII-based.

Okay, makes sense. I hadn't read your earlier posting that carefully before, but it sounds better than what I had in mind.

The question has to do with storage. I suppose you could serialize it and store in a flat file, but will this beat the DB access time, even if there is extra time needed for interpretation?