Python 3.11: “Zero cost” exception handling

Re:Python 3.11: “Zero cost” exception handling by formerly_proven on Wednesday October 06, 2021 @10:36AM (#28773673)

Excellent post! Python Bytecode is a little more naive / high level than this though, so stuff like exceptions and their nested handlers are actually implemented in the VM itself. The VM has essentially a second stack for exception and context manager blocks. The compiled bytecode essentially looks like this:

    SETUP_TRY 10 # address where exceptions will be handled
    some stuff that might explode
    POP_BLOCK
    JUMP 20  # jump over exception handlers
    LOAD exception type  # address 10
    COMPARE  # check if exception type matches
    ... handler for the type
    stuff after the try-except  # address 20

The corresponding source would look like

    try:
        some stuff that might explode
    except exception type:
         ... handler for the type
    stuff after the try-except

Python's bytecode compiler is generally a 1:1 translation of the AST; it never optimizes, e.g:

    [value for value in list]

Translates to something like

    LIST_NEW
    FOR_EACH
    STORE
    LOAD
    LIST_APPEND
    JUMP BACK

Note how "value" generated a store-load pair.

The VM checks whether PyErr (pointer to exception) is set after basically everything. Similarly, extension modules check PyErr after every call into the interpreter, e.g.

    PyObject *attr = PyObject_GetAttr(someattr, somepystr);
    if(PyErr_Occurred()) { // or !attr
        // handle exception
    }

This gets old pretty fast.

[ Parent | on HN ]

Re:Python 3.11: “Zero cost” exception handling by mixedmath on Wednesday October 06, 2021 @12:12PM (#28774830)

I'm growing more interested in actually understanding some of the internals that you mention here. I know this is a bit of a tangent, but is there a better way to approach understanding python's internals than reading the source (which feels a bit monolithic to me right now)?

[ Parent | on HN ]

Re:Python 3.11: “Zero cost” exception handling by formerly_proven on Wednesday October 06, 2021 @12:23PM (#28774959)

The central dispatch of the VM is a good place to start: https://github.com/python/cpython/blob/main/Python/ceval.c#L...

[ Parent | on HN ]

Re:Python 3.11: “Zero cost” exception handling by boothby on Wednesday October 06, 2021 @12:26PM (#28774993)

I thought I had a pretty good handle on Python internals, until some time early this year when I took an interest in the generated bytecode. I'd read plenty of the cpython source, written lots of cython extensions, etc., but somehow missed the middle piece.

Fortunately, it's really easy to get at the bytecode, and quite instructive. Random inquiry: how do generator functions work?

  In [1]: def foo(): 
     ...:     yield from range(10) 
     ...:                                                                                                                                                                                                            

  In [2]: import dis                                                                                                                                                                                                 

  In [3]: dis.dis(foo)                                                                                                                                                                                               
    2           0 LOAD_GLOBAL              0 (range)
                2 LOAD_CONST               1 (10)
                4 CALL_FUNCTION            1
                6 GET_YIELD_FROM_ITER
                8 LOAD_CONST               0 (None)
               10 YIELD_FROM
               12 POP_TOP
               14 LOAD_CONST
               16 RETURN_VALUE

From there, you can read how each of those bytecode instructions is implemented in ceval.c, which formerly_proven links to.

edit: probably nice to have the actual disassembly of a list comprehension, too:

  In [4]: def bar(): 
     ...:     return [x for x in range(10)] 
     ...:                                                                                                                                                                                                            
  
  In [5]: dis.dis(bar)                                                                                                                                                                                               
    2           0 LOAD_CONST               1 (<code object <listcomp> at 0x7f814059f450, file "<ipython-input-4-a7a3cc7e7d7f>", line 2>)
                2 LOAD_CONST               2 ('bar.<locals>.<listcomp>')
                4 MAKE_FUNCTION            0
                6 LOAD_GLOBAL              0 (range)
                8 LOAD_CONST               3 (10)
               10 CALL_FUNCTION            1
               12 GET_ITER
               14 CALL_FUNCTION            1
               16 RETURN_VALUE
  
  Disassembly of <code object <listcomp> at 0x7f814059f450, file "<ipython-input-4-a7a3cc7e7d7f>", line 2>:
    2           0 BUILD_LIST               0
                2 LOAD_FAST                0 (.0)
          >>    4 FOR_ITER                 8 (to 14)
                6 STORE_FAST               1 (x)
                8 LOAD_FAST                1 (x)
               10 LIST_APPEND              2
               12 JUMP_ABSOLUTE            4
          >>   14 RETURN_VALUE

[ Parent | on HN ]

Re:Python 3.11: “Zero cost” exception handling by Chris2048 on Sunday October 10, 2021 @01:13PM (#28820073)

> Python's bytecode compiler is generally a 1:1 translation of the AST; it never optimizes

I thought python had a "peephole optimizer" that optimises?

[ Parent | on HN ]