News for hackers. Stuff that matters.
Re:Say “no” to import side‐effects in Python by Chris_Newton on Saturday April 05, 2014 @11:22AM (#7537305)
Would anyone like to share their experiences avoiding this sort of problem in the context of web frameworks and building the back end for larger web sites/apps?

As an example for discussion, the first time I wrote a Flask-based back-end, I backed myself into a corner almost immediately in the following way.

Firstly, the WSGI file that the web server uses to start the application followed the suggestion in the Flask docs by doing this:

    # webserverseesthis.wsgi
    from yourapplication import app as application
That’s not so bad, but then I started doing application configuration and loading various Flask plug-ins as side effects of that import:

    # yourapplication/__init__.py
    app = Flask("yourapplication")
    
    # Do some general application configuration.
    app.config.from_pyfile("/path/to/configuration/file")

    # Set up some overarching security things that modify application behaviour.
    from flaskext.securityplugin import SecurityPlugin
    sp = SecurityPlugin(app)
This seemed at the time like the obvious place to put such things, but of course, this is really just a variation on the mistake we’re discussing here.

To compound the error, I then used Flask’s decorators to wire up routes from various URLs to the relevant parts of my code. Those decorators work on the application object (sticking with ideas common to many Python web frameworks and avoiding getting into anything more Flask-specific like blueprints) so I was effectively creating circular dependencies from almost everything to that top-level package:

    # yourapplication/pages/home.py
    from yourapplication import app

    @app.route('/')
    def home_page():
        # Render home page
and then from the top-level package onto almost everything so all those decorators could take effect:

    # After setting up the application object in yourapplication/__init__.py
    import yourapplication.pages.home
Now, as long as this kind of code only ever runs as a WSGI application behind a web server, you get away with these dependencies up to a point. In practice, your WSGI set-up imports the top-level application package, which in turn sets up the application object everything is going to depend on and only then imports all the supporting modules/packages, and everything “works”.

However, as soon as you want to write tests or otherwise reuse any of the code in a different context, the entire system is a big bowl of spaghetti with all the usual problems. The moment you import any part of the system to run a unit test on something in it, you get much of the rest of the system as well, complete with the side effects of any imports therein.

This was of course all horribly naïve on general programming principles, but the nature of these frameworks tends to push in this direction, and even Flask’s own documentation features various simple examples that follow a similar approach, so I’ll forgive myself for falling into the trap the first time. I’ve since experimented with various techniques to break the cycles and avoid the side effects on imports, with some success, but frankly I’ve never found a satisfying, general strategy for organising larger code bases built around a web framework.

How is everyone else doing this?

Re:Say “no” to import side‐effects in Python by hcarvalhoalves on Saturday April 05, 2014 @02:55PM (#7538027)
The example on the Flask tutorial with the app at module level is really only viable if you cram everything inside one module, it gets old soon. You should use a factory pattern, like this:

    def app_factory(config):
        app = Flask("yourapplication")
        app.config.from_pyfile(config)
        # ...
        return app
Then whenever you need access to your app object, you use the provided proxy:

    from flask import current_app as app
You can't use it at module level though (because there isn't an application context setup by that time), so this doesn't work:

    @app.route('/')
    def home_page():
        # ...
Instead, hook up views inside your app factory:

    def app_factory(config):
        # ...
        app.route('/')(somemodule.home_page)
        # ...
For the test suite, you can now instantiate apps with a different configuration:

    from flask import current_app as app
    from myfoo import app_factory
    import unittest

    class MyFooTest(unittest.TestCase):
        # ...

    if __name__ == '__main__'
        test_app = app_factory(test_config)
        # The app proxy will point to that inside test cases
        unittest.main()

TL;DR: The factory pattern is your friend. Parametrize all the things. Avoid singletons at module level, this leads to spaghetti. If you need convenience, create proxies.
Re:Say “no” to import side‐effects in Python by Chris_Newton on Saturday April 05, 2014 @09:38PM (#7539268)
You should use a factory pattern, like this: [...] You can't use it at module level though (because there isn't an application context setup by that time), so this doesn't work: [...] Instead, hook up views inside your app factory: [...]

That’s basically what I did on my second iteration. It is an improvement in some respects, particularly breaking the circular dependencies caused by using the decorators on the global application singleton. On the other hand, now you need some variation of God Object that not only imports all your modules that used to have decorators but also knows enough about their internal implementation to set up the routes and things like pre- and post-request logic directly on the application object you get back from the factory.

The next logical step after that then seemed to be having each module/package that contains views or similar logic provide some sort of initialization function that is declared when you import the module and takes an application object as a parameter. Then we can use app.add_url_rule and friends to wire up the various handlers within each package/module but decoupled from any sort of global application object that needs the circular import. This is the tidiest style I’ve found so far, and all my Flask projects in recent years have used something broadly like it. It does only require one import followed by one initialization call for each package/module, which logically seems to be as good as we can get, given that our starting point is a desire to avoid including any initialization implicitly within the import itself and to avoid depending on global singletons.

Somehow, it still doesn’t quite feel right for some reason. I think it’s because even with that general design, I’ve still got a recurring pattern in each of how I create these modules and how I import and then initialize them. My instinct says we ought not to need that extra boilerplate in a highly dynamic language like Python, but I’ve yet to find any alternative that is neater in general. At least in the most simple cases this only adds a couple of extra lines (converting the decorators to an init function in each package/module, and then calling that function at the top level after importing the package/module), which is clearly better than the earlier, more highly connected designs.

Re:Say “no” to import side‐effects in Python by TronPaul_ on Saturday April 05, 2014 @11:40AM (#7537356)
I started running into that problem quickly as well. One of the examples I crib off of is Overholt (http://mattupstate.com/python/2013/06/26/how-i-structure-my-...). It still feels strange in places, but it's the best I've seen so far.
Re:Say “no” to import side‐effects in Python by ategra on Saturday April 05, 2014 @12:09PM (#7537431)
I'm on my mobile, so I can't paste code here. But the way I do it is to subclass the flask app and setup the routes in the setup method. Almost nothing happens on module scope. All views are defined in a seperate module using subclasses of flask views and they hold weak references to the app object. With this setup I neve had any problems testing individual modules. It also works together with flask's unit test client.
Re:Say “no” to import side‐effects in Python by untitaker_ on Saturday April 05, 2014 @12:37PM (#7537527)
App factories help with this a little bit: http://flask.pocoo.org/docs/patterns/appfactories/