* haproxy - frontline proxy
  * varnish - app server cache
  * nginx - static files
  * uwsgi - app server
  * flask - primary framework
  * tornado - async/comet connections
  * bulbflow - graph database toolkit
  * rabbitmq - queue/messaging
  * redis - caching & sessions
  * neo4j - high performance graph database
  * hadoop - data processing

And for the web frame works side, have you found flask to be too light at times?

Flask is at about the right level of abstraction for what a Web framework should be these days. In this era of the social graph, it can be more interesting to store your social graph in a graph database and use it as your primary datastore. And if you're not using a relational database as the primary datastore, why would you want a framework that's built around an ORM?

ORM-based frameworks are ok if you stay inside the box, but they can get in the way when you're not using the RDBMS for authentication and authorization. And when you strip out all the stuff that's tied to the ORM and auth, you end up with with something that looks a lot like Flask. It's usually cleaner to start with something that was designed from the ground up to be a polyglot framework.

P.S. We chose uswgi because it's high performance and high quality (Roberto is a really smart guy), and there's a little-known feature in works -- uwsgi binary connectors to varnish and haproxy that will enable you to connect uwsgi directly to varnish and haproxy over a binary protocol and thus eliminate the HTTP overhead.

Varnish has a modern architecture that's superior to most of the other Web caches (https://www.varnish-cache.org/trac/wiki/ArchitectNotes) -- it was designed by Poul-Henning Kamp, the FreeBSD kernel hacker, and the code is so clean it was published as a book (http://phk.freebsd.dk/misc/_book2.pdf)

Makes me want to try a few of those that are new to me though.

  * puppet - managing server packages / infrastructure
  * monit - monitoring server processes / fixing things
  * django - primary web framework and ORM
  * amazon mysql - it's hosted, and works via plug-ins with Django
  * amazon s3 - storing static assets (images, css, javascript, etc.)
  * amazon elastic load balancer - for scaling incoming HTTP requests across multiple web app servers
  * amazon autoscale - for spinning up new web app servers to handle spikes in traffic
  * rabbitmq - message queueing
  * celery - processing async tasks in a robust fashion. must have
  * memcahed - no explanation necessary
  * git
  * fabric for deploying software
  * jenkins for testing / building software
  * nginx for buffering elastic load balancing requests to web app servers

- uWSGI - performs better than gunicorn and has support for async apps using gevent

- nginix - front end server

- pyramid - web framework

- mongodb - database

- mongoengine - mongodb and python mapper

- zeromq - messaging and communication

- jinja2 - for template engine

- gevent - for async processing

- gevent-zeromq - to make zeromq non-blocking and gevent compatible

- socket-io - JS lib for realtime communication

I still need to develop robust session management. I considered various options and came to conclusion if I want something fast, truly distributed and not using sticky session I should come up with my own session manager demon hosted on each node. I would use ZeroMQ to communicate to it.

I personally use Redis, but you could shove your sessions into Mongodb since you're already using it.

Otherwise my preferred stack is very similar to yours, except I use Mako for templates and PostgreSQL/Redis for backend storage.

So here is how I was considering to build:

- sessions updated and validated by a demon process per node.

- each validation and update will be one call via ZeroMQ's Req/Rep pattern. With each call I can validate session and reset timestamp.

- after each validation I asynchly will replicate session across other nodes via ZeroMQ's Pub/Sub ( I don't care about extra memory)

- sessions will also persist in MongoDB (async), just in case each node is restarted, thus preserving session.

Btw, I only want to valid/invalidate a session token and keep authorization information. Any other small values I could simply keep in the cookies encrypted, e.g. User ID. Though I could keep session information in cookie as well, but this allows sessions to live forever and it's not good if I want to kick some user out.

It's easier to write my own with ZeroMQ and I can build with custom logic at demon level.

I don't have a good grasp of what you're trying to achieve and why, but my overengineering alarms are going off.

But your concern about growing table of sessions is valid, which I will handle by periodically archiving the old sessions so that index sizes remain small.

For extra reliability in MongoDB, I am thinking to use replica sets (active/passive) and journaling.

http://nichol.as/benchmark-of-python-web-servers

Also, the benchmark of Python web servers that gets linked everywhere (http://nichol.as/benchmark-of-python-web-servers) is getting old. I'm planning on doing a new benchmark, probably this coming weekend. As of now, I'm planning to test gunicorn, uWSGI, tornado, bjoern, eventlet, and gevent over HTTP, flup over FCGI, and uWSGI and wsgid over zeroMQ (behind Mongrel2). Thinking of it, I probably need to put all of the HTTP servers behind nginx for a more fair comparison. Am I forgetting any servers that people would like to see benchmarked?

Looking forward, thanks!

  * haproxy - frontline proxy
  * nginx - static files and back proxy
  * supervisord - service uptime
  * gevent/meinheld - wsgi
  * django
  * gevent/eventlet - websockets/comet
  * postgresql - Database obviously
  * memcached - caching for django
  * rabbitmq - message queuing
  * celery - message processing
  * fabric - deploying
  * hudson - building

The only things I would add are "solr" for search, and "redis" for miscellaneous speed improvements, such as statistics tracking and counting.

- ubuntu/debian - apt ftw

- python 3

- haproxy - proxy

- nginx - w/ uwsgi

- cherrypy - framework that supports PY3

- sqlalchemy - orm and sql

- postgres - relational storage

- mongodb - "mandatory" NoSQL

- 0MQ - messaging

Slides and video here: http://lanyrd.com/2011/brightonpy-building-lanyrd/

You won't have to spend much time learning it or fighting with it -- you won't find yourself asking, "Will I be able to do what I want in the framework without hacking it?" Flask let's you program in Python rather than writing to the framework like you typically have to in larger, opinionated framework's like Django and Rails.

Ironically, this also makes Flask an ideal choice for advanced Python programmers because it gives you flexibility rather than always wondering "will the framework allow me to easily do...?"

BTW uwsgi is a production app server. For example nginx has a built-in uwsgi connector and you use uwsgi to serve Flask apps (see http://flask.pocoo.org/docs/deploying/), but it's not a framework like Django/Pyramid/Flask.

Flask is still excellent for small to medium, toy sized projects. Weekend hacks thrown together as a demo, for example. Beyond that I've found the lack of structure and third-party applications in Flask to be a hinderance for more mature sites. I'd say that for any given feature I need on a site that isn't extremely specific to the purpose of it, 90% of the time I can find it on django-packages and have it dropped-in and integrated within 10-15 minutes.

I'd venture to guess that for every hour I spend getting something to work that doesn't fall nicely into Django's structure, I save a hundred or so from not having to re-write a mature component that the community has solved ten times over.

My preferred setup that works for most cases. All reliable and fast.

we use varnosh as a frontend server it handles the load balancing betwen our UWSGI servers, and if the request is a static file its send to our nginx server. we them use redis to store all of our cache and sessions, we cache everything so everytime there is a read from our database via the django ORM our api grabs the whole object returned and stores it in redis so next time we need to retrieve it we just hit redis.

tornado - web

memcache - cache

mysql - database

My current project requires real-time/always-on connections so I started to develop it using Tornado and then decided to switch to the Quora model -- use a traditional Web framework like Flask for most things, and connect back to Tornado for the real-time stuff.

When I switched to this model, the development process sped up considerably. In addition to just being really-well designed, Flask has an amazing debugger that makes me more productive, and it's easier to write unittests for Flask because you can write them in a traditional way and don't have to contend with Tornado's IOLoop.

For real-time stuff, you could forgo Tornado all together and instead use gevent to deploy your Flask app (http://flask.pocoo.org/docs/deploying/others/#gevent), like some have done with Django and Pyramid, but I haven't tried this yet.

Personally, I'm happy with eventlet and can't see why I would want to go to gevent. Does anybody using gevent want to chime in on what's better about it?

In your case you could switch to a pure Python MySQL driver and gevent will turn all your mySQL calls into async.

Actually there're also gevent non-blocking MySQL drivers that are written in C.

More seriously, Twisted/Flask/SQLAlchemy has been the formula for the past two deployments I've done, and I'm happy with it.

In other word, with Twisted, you can write http, ftp, smtp, ... and your are free to write your own protocol.

~ Mature. Twisted has been around for a lot longer than Tornado and has learned from all of the available history of modern OS networking.

~ Tested. Twisted's been in production for a long, long time, and is covered with thousands of unit tests. It's official policy that code may not enter the Twisted tree without accompanying tests.

~ Flexible. Twisted can be used as a general-purpose networking library, it can integrate with Pygame and Pyglet, GTK+, Wx, Qt, Tk; it doesn't have to be used for servers.

~ Extensible. Twisted's connectors are explicit, and rely on interfaces and adapters rather than inheritance. As an example, Twisted's SSH library lets you separate the SSH server, SSH channel, and SSH shell from each other. Annoying if you want a standard SSH server, but terrific if you're building a custom SSH proxy or tunnel. (I did this a few weeks ago at work. A lot easier with Twisted than with Paramiko!)

I should note that it's not an either-or; there is a branch of Tornado which throws out all of the event loop and uses Twisted's event loop instead.

The tornado and cyclone APIs are closer to what I am thinking about as a web developer. Virtually never do I make an object and then think about how I'd like to adapt it to return an HTML view or programmatically add it under a parent object.