Re:Applying the Unix Philosophy to Object-Oriented Design
by nuxi on Thursday November 29, 2012 @03:46AM
(#4846807)
(Minor nit: the yield-based functions are generator functions, not iterators.)
This approach was also described by David Beazley at PyCon, the slides are available at http://www.dabeaz.com/generators/Generators.pdf. He extends this approach to generator multiplexing, coroutines etc., along with useful examples. An excellent read.
I primarily work on a Python codebase, and I've found that using iterators for complex, fault-tolerant data pipelines allows decoupled design without many of the performance & additional complexity drawbacks often encountered with cleanly abstracted, decoupled code. For example, when executing a multi-get for objects by primary key, the pipeline looks roughly like this:
1. Fetch from heap cache
2. Fetch from remote cache
3. Fetch from backing store
4. Backfill the heap cache
5. Backfill the remote cache
6. Apply basic filters (e.g. deleted == False, etc)
At each step there are usually two or three layers of abstraction underneath. Much of the space requirements, and some of the overhead time at each step can be collapsed to O(1) instead of O(N).
For example, a cache multiget abstraction on-top of memcache might look something like this:
Notice how at each step there is a large amount of "buffering" that causes allocation, copying, and quite a bit of additional work. Each layer of abstraction adds a pretty large cost to the step. Using an iterator implementation, we can clean up this code and make it more performant: It's clear how much cleaner this code is. Notice how this snippet avoids the large amount of "buffering" of data between steps and short-circuits quite a bit of code when possible (for instance, if all of the fetches miss). In real code that's heavily abstracted & layered, avoiding all of this work translates into significant performance & cost advantages.Iterators also allow building portions of the pipeline with built-in functions that avoid the interpeter.
This block of code executes in only one pass through the interpreter (imap, itemgetter, and pickle.loads are all implemented as native functions). This is incredibly powerful because it means that iterator-based abstractions can be built by combining these native building blocks without the overhead of recursion within the interpreter at each step.Pushing all the recursion into native code:
* I apologize for any code errors, this code wasn't tested in it's entirety.