Ask HN: What is your preferred Python stack for high traffic webservices?

Re:Ask HN: What is your preferred Python stack for high traf... by antimora on Sunday August 21, 2011 @11:26PM (#2911177)

Thanks. Actually I did consider Beaker it didn't fulfill my requirement. I wanted to replicate sessions across nodes actively and async, and also I wanted to persist to MongoDB async.

So here is how I was considering to build:

- sessions updated and validated by a demon process per node.

- each validation and update will be one call via ZeroMQ's Req/Rep pattern. With each call I can validate session and reset timestamp.

- after each validation I asynchly will replicate session across other nodes via ZeroMQ's Pub/Sub ( I don't care about extra memory)

- sessions will also persist in MongoDB (async), just in case each node is restarted, thus preserving session.

Btw, I only want to valid/invalidate a session token and keep authorization information. Any other small values I could simply keep in the cookies encrypted, e.g. User ID. Though I could keep session information in cookie as well, but this allows sessions to live forever and it's not good if I want to kick some user out.

[ Parent | on HN ]

Re:Ask HN: What is your preferred Python stack for high traf... by bigethan on Sunday August 21, 2011 @11:29PM (#2911182)

What's the use case that raises these requirements? Or, more politely, what problem are you solving with this setup?

[ Parent | on HN ]

Re:Ask HN: What is your preferred Python stack for high traf... by antimora on Monday August 22, 2011 @12:09AM (#2911259)

Primarily I wanted it to be really fast, be able to invalidate session, and be available on any node via active replication. If I used Beaker, I had to use memcached which is not replicated (by default setup), and each validation would have required 2 round trips to memchached (validation, update stamp).

It's easier to write my own with ZeroMQ and I can build with custom logic at demon level.

[ Parent | on HN ]

Re:Ask HN: What is your preferred Python stack for high traf... by shazow on Monday August 22, 2011 @02:10AM (#2911527)

You don't have to use memcached, the beaker extensions let you use whichever backend storage mechanism you want. Redis Cluster can be replicated and would work fine for this, for example.

I don't have a good grasp of what you're trying to achieve and why, but my overengineering alarms are going off.

[ Parent | on HN ]

Re:Ask HN: What is your preferred Python stack for high traf... by antimora on Monday August 22, 2011 @02:27AM (#2911573)

Thanks for the suggestion. I haven't considered Redis Cluster, but I definitely will.

[ Parent | on HN ]

Re:Ask HN: What is your preferred Python stack for high traf... by espeed on Monday August 22, 2011 @01:31AM (#2911436)

Awhile back I looked into Mongo for sessions, but it isn't really designed to do this. Sessions are temporal, and AFAIK Mongo doesn't have a built in mechanism for expiring them, plus session data is usually small -- there are other datastores better suited for this type of data.

[ Parent | on HN ]

Re:Ask HN: What is your preferred Python stack for high traf... by antimora on Monday August 22, 2011 @02:10AM (#2911525)

Primarily sessions will live in RAM. And I was going to use MongoDB to persist sessions just in case the daemon dies or session is not found in RAM as a fallback. The reason for using MongoDB is because it's my primary database.

But your concern about growing table of sessions is valid, which I will handle by periodically archiving the old sessions so that index sizes remain small.

[ Parent | on HN ]