The example makes sense when you think of `i` getting reassigned, but in other languages with variable shadowing (think Rust, but Go too when you're in a nested scope) you might replace `i = 2` with `let i = 2` (Rust), in which case the closure closes over precisely the `let i = 2` variable and the expected output differs.

I think the reason this seems surprising is the value/reference semantics of values. For example:

    def foo(i, j):
      i[0] += 1
      j += 1
    a = [0]
    b = 0
    foo(a, b)
    print(a) # [1]
    print(b) # [0]

So people think the body of the loop like a function call.

I don't think it's a bad expectation, in fact I think it's quite a natural expectation—in particular if you've programmed functional languages where modifying values is the exception, not the rule—which is why it surprises people. It's just not the one way Python chose.

I think it comes down to if you think it makes sense for variables to be captured by value or by reference in closures.

Which ever makes the most sense depends on the context of a program which is why in languages that have both reference and closure semantics you can choose how the variables are captured. When that isn't the case you need to pick for someone, and it gets weird.

> I think it comes down to if you think it makes sense for variables to be captured by value or by reference in closures.

No, this comes down to “should a loop control variable be scoped to the block—or in python’s case function—the loop is in and updated with each iteration or a fresh variable scoped to each loop iteration that happens to share the same name.”

In the "the variable lives in the function scope", the answer is unequivocally "it should be bound once and updated", if it only exists in the scope of the loop function, both "it is bound on each iteration" (and thus safe to close over without surprise) and "it is bound once and updated" are valid answers, but I have a preference for the first, but many languages actually choose the second.

I feel that the original Python example is suffering from a little too much lambda.

It seems like we're iterating through something to build up a computation we may execute later. I can conceive of a situation where you might want to do that, or at least consider it, but in general I say just do the work now and build a list of results.

It looks like that because it's a minimal example. Real cases usually involve callbacks or expensive computations. How often it happens depends a lot on what you're doing.

> I thought that was rare

It is rare, if only because defining functions (lambdas or otherwise) in a loop is somewhat uncommon.

That's not very helpful when you hit that issue though.

That's one way of thinking about it, and it works well for Python.

Here's another one, and the way I think about it:

    def _loop(i):
        functions.append(lambda: print(f"Hello {i}"))

    _loop(0)
    _loop(1)
    _loop(2)

    print(i)
    # Error!

I highly prefer lexical scoping, where the variables are bound to the block they were declared in, like Javascript's `let` vs the old `var`. This avoids shadowing and general namespace pollution.

I know it's not how Python operates, but I think it's how it should. Though I'd argue the syntactical similarities between functions and loops nudge users towards this second model.

Note that the way you write this is the best way to form closures in Python: form them inside a function with the closure variables as function parameters. This forces the closure to capture the current value of the closure variables when the closure is formed. Note that you could put the calls to the _loop function in a for loop and things would still work:

    >>> functions = []
    >>> def _loop(i):
    ...     functions.append(lambda: print(f"Hello {i}"))
    ...
    >>> for i in range(10):
    ...     _loop(i)
    ...
    >>> for f in functions:
    ...     f()
    ...
    Hello 0
    Hello 1
    Hello 2
    Hello 3
    Hello 4
    Hello 5
    Hello 6
    Hello 7
    Hello 8
    Hello 9

> lexical scoping, where the variables are bound to the block they were declared in

In Python that would only work if a "block" included comprehensions. For example:

    >>> functions = [lambda: print(f"Hello {i}") for i in range(10)]
    >>> for f in functions:
    ...     f()
    ...
    Hello 9
    Hello 9
    Hello 9
    Hello 9
    Hello 9
    Hello 9
    Hello 9
    Hello 9
    Hello 9
    Hello 9

You could fix this by defining a _loop function as above and forming the closure inside it; but changing variable scoping to be lexical in "blocks" wouldn't fix this case unless the list comprehension itself counted as a "block", which is not how Python defines blocks.

Wow. When I saw the original for-loop example I was like "ok not a huge deal, doesn't surprise me too much really". But rewritten in the comprehension form I'm like wtf

Well put. I too prefer the reasonability of lexical scoping. Although I must confess I am not well versed enough to quite grasp how what Python does _isn't_ conforming to lexical scope.

Since I was confused I did some internet searching and found this:

https://stackoverflow.com/questions/51604346/does-python-sco...

Which I think suggests the thing you are finding confusing isn't lexical scoping in Python, but rather the environment mutability.

Ultimately, I think I better understand what you are highlighting and can't say I entirely disagree. I just personally find how it is today ergonomic, but that could also just be bias as I am fairly comfortable in Python (warts and all).

I woudl say taht Python has scoping, yes, but it does not have lexical scoping in any sense of "lexical scoping" that I am aware of. If it did, the code below would not actually work ,as the outside-the-loop print would be trying to access a variable not available in the lexical scope of "the function body", as it is defined and established within the lexical scope of "the loop".

So, at least in my book, no, Python has "global scope", "function scope" and probably one or two more scopes (I think there's a "class scope" as well).

Here's some code in Python, and some equivalent code in Go.

    def foo(a_list):
        print(f"list is {len(a_list} elements")

        for element in a_list:
            print(element)
        print(element)

And here's the equivalent Go code:

    func foo(aList []int) { // Let's use ints...
        fmt.Printf("list is %d elements\n", len(aList))
        var element int // Notice this declaration! This is ensuring that element is declared outside the lexical scope of the for loop
        for _, element = range aList {
            fmt.Println(element)
        }
        fmt.Println(element)
    }

I think you are confusing type of scoping (lexical/static, dynamic) with scoping levels.

Python has lexical scoping, but it does _not_ have block level scoping.

https://en.wikipedia.org/wiki/Scope_(computer_science)#Block...

Which, now that I have done enough reading I think crystallizes the confusing thing for others being highlighted here (for me). Depending on preference, the lack of block scoping can be surprising for someone. Which also explains my bias, I started with Python which probably plays a large part in why I find function level scoping without block scoping ergonomic.

The problem is the intersection of people's intuition about closures from languages without mutable state bumping into the imperative world built of mutable state.

For example, equivalent code in Elixir:

    i = 0
    list = []
    list = list ++ [fn -> i end] 
    i = i + 1
    list = list ++ [fn -> i end]
    i = i + 1
    halfway = list
    list = list ++ [fn -> i end]
    i = i + 1
    list = list ++ [fn -> i end]
    IO.inspect Enum.map(halfway, fn f -> f.() end)
    IO.inspect Enum.map(list, fn f -> f.() end)

Would produce the functional-intuitive result of [0,1] [0,1,2,3]

Because there is no mutable state. Those repeated assignments to i and list are exactly equivalent to the scenario where each i was actually i1, i2, i3, etc.

I expect that to write the obvious result (the last value assigned) and I also expect a capture in a loop to write the value as it was at the time of iteration.

They are both the behavior that’s intuitively obvious. And that’s despite them both being the same behavior.

As we can see, it’s important to not use the simple/obvious implementation because it’s so unintuitive it’ll need to be changed even if breaking (as in C#)

seems like the non magic solution here would be block scope. I guess it would still be slightly magic in that each iteration gets its own scope, but at least that's easier to wrap your head around than just a special case