News for hackers. Stuff that matters.
Re:Go: Redefining For Loop Variable Semantics by BoppreH on Tuesday October 11, 2022 @07:31AM (#33162048)
The same problem frequently bites me in Python too:

    items = []

    for i in range(10):
        items.append(lambda: print(f'Hello {i}'))
    
    for item in items:
        item()

    # Hello 9
    # Hello 9
    # Hello 9
    # Hello 9
    # Hello 9
    # Hello 9
    # Hello 9
    # Hello 9
    # Hello 9
    # Hello 9
I find it one of Python's biggest warts because it's silent, hard to troubleshoot (especially the first time!), and the most straightforward fix looks ridiculous (i=i):

    for i in range(10):
        items.append(lambda i=i: print(f'Hello {i}'))
I guess this is a lesson in designing language semantics that match people's intuitions, and learning from previous languages' mistakes.
Re:Go: Redefining For Loop Variable Semantics by Simplicitas on Tuesday October 11, 2022 @08:07AM (#33162366)
Yup. "Match [human] intuition" is the operative phrase in all of this. Thanks. Hope Go addresses this soon.
Re:Go: Redefining For Loop Variable Semantics by Semaphor on Tuesday October 11, 2022 @09:21AM (#33163035)
If I do the same with for instead of foreach in C#, my IDE gives me a warning: "Captured variable is modified in the outer scope". Isn’t that the case for Python?
Re:Go: Redefining For Loop Variable Semantics by ketzu on Tuesday October 11, 2022 @09:43AM (#33163261)
Copy pasting OPs code to pycharm: No warning.
Re:Go: Redefining For Loop Variable Semantics by Semaphor on Tuesday October 11, 2022 @09:47AM (#33163309)
Probably worthy of a YouTrack report ;)
Re:Go: Redefining For Loop Variable Semantics by jstx1 on Tuesday October 11, 2022 @10:46AM (#33163970)
Re:Go: Redefining For Loop Variable Semantics by maxloh on Tuesday October 11, 2022 @02:27PM (#33167078)
I don't think it is a good practice to capture it in lint tooling, instead of by the language itself.
Re:Go: Redefining For Loop Variable Semantics by Semaphor on Tuesday October 11, 2022 @10:18PM (#33171964)
It’s what linters are great at, catch things that work the way you don’t expect.
Re:Go: Redefining For Loop Variable Semantics by throw_m239339 on Tuesday October 11, 2022 @09:24AM (#33163057)
This is mostly why Ecmascript introduced lexical scoping (a bit similar to "block" scoping) with "let" back int ES 5 I believe, because var used the function scope and anymore would run into weird isues whenever they did this:

     for (var i = 0 ; i < 10 ; i ++ ){
         someElement.addEventListener('click',function(){
            console.log(i);
         })
     }

     
i would always be equal to 9. With let instead of var, i is properly scoped and the executed script display each increment correctly.

The solution before let was to introduce a closure in the for statement body

     for (var i = 0 ; i < 10 ; i ++ ){
         (function(i){
             someElement.addEventListener('click',function(){
                 console.log(i);
         }))(i)
     }
in order to capture i value. let can also easily isolate the scope of a variable so that it doesn't popule the global scope

    {
       let foo = "bar";
       var baz = "qix";
    }
    
    // foo is undefined here, while baz is defined.
which removes the need for self invoking functions.
Re:Go: Redefining For Loop Variable Semantics by gshulegaard on Tuesday October 11, 2022 @09:48AM (#33163315)
I found this interesting because I have never run into this before...and I thought that was rare (in relation to how comfortable I am with Python). That said, I personally think this is the right and intuitive outcome.

If we expand out the loop to be manual we would have a script like:

   functions = []
   
   # expanded loop
   i = 1
   functions.append(lambda: print(f"Hello {i}"))
   i = 2
   functions.append(lambda: print(f"Hello {i}"))
   ...
   i = 9
   functions.append(lambda: print(f"Hello {i}"))
Now at this point if we were to:

   print(f"Hello {i}")
What would the expected output be? I would posit that anything other than "Hello 9" would be wrong, both logically and intuitively.

So by extension a loop of effectively "print(f"Hello {i}")" 9 times should just print "Hello 9" 9 times IMO. Anything else is counter-intuitive and definitely surprising.

Re:Go: Redefining For Loop Variable Semantics by tmazeika on Tuesday October 11, 2022 @10:33AM (#33163834)
The example makes sense when you think of `i` getting reassigned, but in other languages with variable shadowing (think Rust, but Go too when you're in a nested scope) you might replace `i = 2` with `let i = 2` (Rust), in which case the closure closes over precisely the `let i = 2` variable and the expected output differs.
Re:Go: Redefining For Loop Variable Semantics by _flux on Tuesday October 11, 2022 @10:28AM (#33163775)
I think the reason this seems surprising is the value/reference semantics of values. For example:

    def foo(i, j):
      i[0] += 1
      j += 1
    a = [0]
    b = 0
    foo(a, b)
    print(a) # [1]
    print(b) # [0]
So people think the body of the loop like a function call.

I don't think it's a bad expectation, in fact I think it's quite a natural expectation—in particular if you've programmed functional languages where modifying values is the exception, not the rule—which is why it surprises people. It's just not the one way Python chose.

Re:Go: Redefining For Loop Variable Semantics by duped on Tuesday October 11, 2022 @10:48AM (#33163992)
I think it comes down to if you think it makes sense for variables to be captured by value or by reference in closures.

Which ever makes the most sense depends on the context of a program which is why in languages that have both reference and closure semantics you can choose how the variables are captured. When that isn't the case you need to pick for someone, and it gets weird.

Re:Go: Redefining For Loop Variable Semantics by dragonwriter on Tuesday October 11, 2022 @01:03PM (#33165815)
> I think it comes down to if you think it makes sense for variables to be captured by value or by reference in closures.

No, this comes down to “should a loop control variable be scoped to the block—or in python’s case function—the loop is in and updated with each iteration or a fresh variable scoped to each loop iteration that happens to share the same name.”

Re:Go: Redefining For Loop Variable Semantics by randomswede on Wednesday October 12, 2022 @03:04AM (#33173794)
In the "the variable lives in the function scope", the answer is unequivocally "it should be bound once and updated", if it only exists in the scope of the loop function, both "it is bound on each iteration" (and thus safe to close over without surprise) and "it is bound once and updated" are valid answers, but I have a preference for the first, but many languages actually choose the second.
Re:Go: Redefining For Loop Variable Semantics by jrumbut on Tuesday October 11, 2022 @10:50AM (#33164013)
I feel that the original Python example is suffering from a little too much lambda.

It seems like we're iterating through something to build up a computation we may execute later. I can conceive of a situation where you might want to do that, or at least consider it, but in general I say just do the work now and build a list of results.

Re:Go: Redefining For Loop Variable Semantics by BoppreH on Tuesday October 11, 2022 @11:36AM (#33164646)
It looks like that because it's a minimal example. Real cases usually involve callbacks or expensive computations. How often it happens depends a lot on what you're doing.
Re:Go: Redefining For Loop Variable Semantics by masklinn on Tuesday October 11, 2022 @11:10AM (#33164283)
> I thought that was rare

It is rare, if only because defining functions (lambdas or otherwise) in a loop is somewhat uncommon.

That's not very helpful when you hit that issue though.

Re:Go: Redefining For Loop Variable Semantics by BoppreH on Tuesday October 11, 2022 @11:21AM (#33164425)
That's one way of thinking about it, and it works well for Python.

Here's another one, and the way I think about it:

    def _loop(i):
        functions.append(lambda: print(f"Hello {i}"))

    _loop(0)
    _loop(1)
    _loop(2)

    print(i)
    # Error!
I highly prefer lexical scoping, where the variables are bound to the block they were declared in, like Javascript's `let` vs the old `var`. This avoids shadowing and general namespace pollution.

I know it's not how Python operates, but I think it's how it should. Though I'd argue the syntactical similarities between functions and loops nudge users towards this second model.

Re:Go: Redefining For Loop Variable Semantics by pdonis on Tuesday October 11, 2022 @11:41AM (#33164720)
Note that the way you write this is the best way to form closures in Python: form them inside a function with the closure variables as function parameters. This forces the closure to capture the current value of the closure variables when the closure is formed. Note that you could put the calls to the _loop function in a for loop and things would still work:

    >>> functions = []
    >>> def _loop(i):
    ...     functions.append(lambda: print(f"Hello {i}"))
    ...
    >>> for i in range(10):
    ...     _loop(i)
    ...
    >>> for f in functions:
    ...     f()
    ...
    Hello 0
    Hello 1
    Hello 2
    Hello 3
    Hello 4
    Hello 5
    Hello 6
    Hello 7
    Hello 8
    Hello 9
> lexical scoping, where the variables are bound to the block they were declared in

In Python that would only work if a "block" included comprehensions. For example:

    >>> functions = [lambda: print(f"Hello {i}") for i in range(10)]
    >>> for f in functions:
    ...     f()
    ...
    Hello 9
    Hello 9
    Hello 9
    Hello 9
    Hello 9
    Hello 9
    Hello 9
    Hello 9
    Hello 9
    Hello 9
You could fix this by defining a _loop function as above and forming the closure inside it; but changing variable scoping to be lexical in "blocks" wouldn't fix this case unless the list comprehension itself counted as a "block", which is not how Python defines blocks.
Re:Go: Redefining For Loop Variable Semantics by dkmar on Tuesday October 11, 2022 @05:24PM (#33169424)
Wow. When I saw the original for-loop example I was like "ok not a huge deal, doesn't surprise me too much really". But rewritten in the comprehension form I'm like wtf
Re:Go: Redefining For Loop Variable Semantics by gshulegaard on Tuesday October 11, 2022 @11:53PM (#33172626)
Well put. I too prefer the reasonability of lexical scoping. Although I must confess I am not well versed enough to quite grasp how what Python does _isn't_ conforming to lexical scope.

Since I was confused I did some internet searching and found this:

https://stackoverflow.com/questions/51604346/does-python-sco...

Which I think suggests the thing you are finding confusing isn't lexical scoping in Python, but rather the environment mutability.

Ultimately, I think I better understand what you are highlighting and can't say I entirely disagree. I just personally find how it is today ergonomic, but that could also just be bias as I am fairly comfortable in Python (warts and all).

Re:Go: Redefining For Loop Variable Semantics by randomswede on Wednesday October 12, 2022 @03:02AM (#33173779)
I woudl say taht Python has scoping, yes, but it does not have lexical scoping in any sense of "lexical scoping" that I am aware of. If it did, the code below would not actually work ,as the outside-the-loop print would be trying to access a variable not available in the lexical scope of "the function body", as it is defined and established within the lexical scope of "the loop".

So, at least in my book, no, Python has "global scope", "function scope" and probably one or two more scopes (I think there's a "class scope" as well).

Here's some code in Python, and some equivalent code in Go.

    def foo(a_list):
        print(f"list is {len(a_list} elements")

        for element in a_list:
            print(element)
        print(element)
And here's the equivalent Go code:

    func foo(aList []int) { // Let's use ints...
        fmt.Printf("list is %d elements\n", len(aList))
        var element int // Notice this declaration! This is ensuring that element is declared outside the lexical scope of the for loop
        for _, element = range aList {
            fmt.Println(element)
        }
        fmt.Println(element)
    }
Re:Go: Redefining For Loop Variable Semantics by gshulegaard on Wednesday October 12, 2022 @10:56AM (#33177903)
I think you are confusing type of scoping (lexical/static, dynamic) with scoping levels.

Python has lexical scoping, but it does _not_ have block level scoping.

https://en.wikipedia.org/wiki/Scope_(computer_science)#Block...

Which, now that I have done enough reading I think crystallizes the confusing thing for others being highlighted here (for me). Depending on preference, the lack of block scoping can be surprising for someone. Which also explains my bias, I started with Python which probably plays a large part in why I find function level scoping without block scoping ergonomic.

Re:Go: Redefining For Loop Variable Semantics by OkayPhysicist on Tuesday October 11, 2022 @04:19PM (#33168713)
The problem is the intersection of people's intuition about closures from languages without mutable state bumping into the imperative world built of mutable state.

For example, equivalent code in Elixir:

    i = 0
    list = []
    list = list ++ [fn -> i end] 
    i = i + 1
    list = list ++ [fn -> i end]
    i = i + 1
    halfway = list
    list = list ++ [fn -> i end]
    i = i + 1
    list = list ++ [fn -> i end]
    IO.inspect Enum.map(halfway, fn f -> f.() end)
    IO.inspect Enum.map(list, fn f -> f.() end)
Would produce the functional-intuitive result of [0,1] [0,1,2,3]

Because there is no mutable state. Those repeated assignments to i and list are exactly equivalent to the scenario where each i was actually i1, i2, i3, etc.

Re:Go: Redefining For Loop Variable Semantics by alkonaut on Tuesday October 11, 2022 @04:59PM (#33169167)
I expect that to write the obvious result (the last value assigned) and I also expect a capture in a loop to write the value as it was at the time of iteration.

They are both the behavior that’s intuitively obvious. And that’s despite them both being the same behavior.

As we can see, it’s important to not use the simple/obvious implementation because it’s so unintuitive it’ll need to be changed even if breaking (as in C#)

Re:Go: Redefining For Loop Variable Semantics by asddubs on Tuesday October 11, 2022 @05:07PM (#33169265)
seems like the non magic solution here would be block scope. I guess it would still be slightly magic in that each iteration gets its own scope, but at least that's easier to wrap your head around than just a special case
Re:Go: Redefining For Loop Variable Semantics by jmconfuzeus on Tuesday October 11, 2022 @10:00AM (#33163463)
Technically, it makes sense for Python to work like this because your functions are being bound to a variable i which is located in the global scope.

I wouldn't call this a design mistake but more of a misunderstanding of how scope works.

Re:Go: Redefining For Loop Variable Semantics by remram on Tuesday October 11, 2022 @12:37PM (#33165464)
If other languages get this right then it's not a fundamental misunderstanding but a quirk of that specific language.
Re:Go: Redefining For Loop Variable Semantics by jmconfuzeus on Tuesday October 11, 2022 @12:53PM (#33165691)
Dunno what you mean by getting it right tbh.

for i in range(10)

The statement literally declares or sets a variable named i in that scope. When the loop exits, i still exists in the scope with the value 9. If you call a function that was given a reference to i, the value will be 9 as expected because the function was called after the loop exited.

Don't see any quirk or mistake here.

Re:Go: Redefining For Loop Variable Semantics by masklinn on Tuesday October 11, 2022 @01:23PM (#33166083)
An equivalent construct works differently (and one would say in a way that is less error-prone) in other languages.

For instance the behaviour of a Python loop varies drastically depending on the size of the iteration:

     def loop(n):
         for i in range(n):
             pass
         print(i)

     loop(10) # 9
     loop(0)
     Traceback (most recent call last):
       File "<stdin>", line 1, in <module>
       File "<stdin>", line 4, in loop
     UnboundLocalError: local variable 'i' referenced before assignment
That Python works this way is specific to Python. And a language which doesn't have the issues this implies would be "getting it right", in the sense of avoiding sharp corners and edge cases.
Re:Go: Redefining For Loop Variable Semantics by jmconfuzeus on Tuesday October 11, 2022 @01:39PM (#33166303)
I see, it makes sense to force people to use different variables for iteration in that case because it can easily trip inexperienced developers up.
Re:Go: Redefining For Loop Variable Semantics by int_19h on Tuesday October 11, 2022 @04:54PM (#33169110)
The statement can mean whatever the language designers decide it should mean, and one plausible meaning is to introduce a new scope for the body of the loop. It's not even something unique in Python, given that sequence comprehensions do just that; e.g. this prints 0,1,...,9:

   for f in (lambda: print(i) for i in range(0, 10)): f()
Unfortunately, Python is simply inconsistent in this regard. For example, list comprehensions leak the variable for back-compat reasons, so if you substitute (lambda: ...) with [lambda: ...] above, you'll get a bunch of 9s.

But, backwards compatibility aside, the language could change to make for-loops behave like sequence comprehensions wrt scoping.

Re:Go: Redefining For Loop Variable Semantics by dragonwriter on Tuesday October 11, 2022 @05:03PM (#33169217)
> Unfortunately, Python is simply inconsistent in this regard. For example, list comprehensions leak the variable for back-compat reasons

No, they don’t. They did in Python 2—list comps were introduced in 2.0, genexps in 2.4, and set/dict comps in 3.0 but also included in the later 2.7 release—but that’s been non-current for more than a decade, and conpletely out of support for two years. Let it go.

> But, backwards compatibility aside, the language could change to make for-loops behave like sequence comprehensions wrt scoping.

Sure in Python 4, but after 2->3, not sure many people are looking forward to that.

Re:Go: Redefining For Loop Variable Semantics by int_19h on Wednesday October 12, 2022 @05:31AM (#33174677)

   Python 3.10.5 (tags/v3.10.5:f377153, Jun  6 2022, 16:14:13) [MSC v.1929 64 bit (AMD64)] on win32
   Type "help", "copyright", "credits" or "license" for more information.
   >>> [f() for f in [lambda: i for i in range(0, 10)]]
   [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
   >>> [f() for f in (lambda: i for i in range(0, 10))]
   [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Re:Go: Redefining For Loop Variable Semantics by remram on Wednesday October 12, 2022 @09:45AM (#33176879)
I don't know what this is supposed to prove, it just shows that the generator is lazy while the list isn't. It seems unrelated to scoping issues or leaking variables.
Re:Go: Redefining For Loop Variable Semantics by int_19h on Wednesday October 12, 2022 @03:58PM (#33182219)
It shows that list comprehension has a single loop variable across all loop iterations that gets reassigned on each iteration, while the sequence comprehension creates a new loop variable bound to current item on every iteration.
Re:Go: Redefining For Loop Variable Semantics by ekiru on Wednesday October 12, 2022 @04:31PM (#33182665)
I don't think that's what's happening. In your example with the generator expression, you're calling each lambda as you iterate through the generator, which due to the lazy evaluation of the generator means that the value of the single i variable shared across all each lambda is still only the latest value reached.

If you instead fully evaluate the generator expression before calling any of the functions (for example, by passing it to the list constructor), you get the same behavior as the list comprehension case:

    >>> [f() for f in list(lambda: i for i in range(0, 10))]
    [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
Re:Go: Redefining For Loop Variable Semantics by int_19h on Thursday October 13, 2022 @07:57PM (#33198031)
You're right, but that means that sequence comprehension also leaks the variable, so it's even worse than I thought.

Side note: I think that commenters above didn't quite understand what I meant by "leaking", because there's more than one scope boundary here. Roughly speaking, any comprehension or loop can be desugared into something that looks like a C-style for-loop:

   /* scope 1 */
   for (/* scope 2 */) {
      /* scope 3 */
   }
Scope 1 is outside relative to the loop. Scope 2 is specific to the loop but shared by all its iterations. Scope 3 is specific to one loop iteration. The "leaking" I referred to above is from scope 3 to scope 2. I think other commenters took it to mean leaking from scope 2 to scope 1 - i.e. the ability to use the variable outside of the comprehension; that is, indeed, something that changed between Python 2 and 3.
Re:Go: Redefining For Loop Variable Semantics by Epokhe on Tuesday October 11, 2022 @10:41AM (#33163917)
This seems like a cleverly crafted satire reply, rather than an actual complaint about python.
Re:Go: Redefining For Loop Variable Semantics by iudqnolq on Tuesday October 11, 2022 @11:40AM (#33164706)
If so, it's a bit misleading. The go bug just requires a reference, rather than a lambda. I bucket lambdas into a category of language features I expect to have more sharp edges around capture semantics than references.

Bug:

    var all []*Item
    for _, item := range items {
     all = append(all, &item)
    }
Fix:

    var all []*Item
    for _, item := range items {
     item := item
     all = append(all, &item)
    }
Re:Go: Redefining For Loop Variable Semantics by BoppreH on Tuesday October 11, 2022 @02:43PM (#33167372)
I assure you it's not satire, otherwise it wouldn't be such a popular question on StackOverflow[1] and an item in the Python FAQ[2]

[1]: https://stackoverflow.com/questions/2295290/what-do-lambda-f...

[2]: https://docs.python.org/3/faq/programming.html#why-do-lambda...

Re:Go: Redefining For Loop Variable Semantics by Epokhe on Tuesday October 11, 2022 @06:32PM (#33170116)
Yeah, makes sense that it's popular. I just thought you were making a joke by creating a similar looking issue with a different underlying reason.

In Go example the issue happens because item variable is per-loop, in your python example the issue is not related to loops at all, it's just because functions capture the value of global variables at execution time.

And the cherry on top is that the solution is also similar looking(i=i), but working with a different mechanic underneath(default argument assignment).

Anyway, this was my perspective that led me to interpret this as satire. A bit disappointed haha

Re:Go: Redefining For Loop Variable Semantics by dekhn on Tuesday October 11, 2022 @11:09AM (#33164266)
I guess this is confusing but i just think of it as capturing the reference to i, not the value of i. It would be nice if python had a nice way to deal with this as many people also trip over:

def fun(initial_empty_list=[]): where initial_empty_list is a reference captured at function definition time, not a new value initialized on each call to the function.

Re:Go: Redefining For Loop Variable Semantics by littlestymaar on Tuesday October 11, 2022 @01:25PM (#33166099)
Wow, Python and Go have it too? It was the biggest wart of pre-ES6 JavaScript, I never imagined other languages had it too (and it's honestly very disappointing for Go, since it's much more recent and we had plenty of hindsight when it was created…)

Edit: looks like Rust does it right: https://play.rust-lang.org/?version=stable&mode=debug&editio...

Re:Go: Redefining For Loop Variable Semantics by kevincox on Tuesday October 11, 2022 @03:00PM (#33167631)
Not quite how you tested it. The `move` is actually copying the variable:

    fn main() {
        let mut functions = vec![];
    
        let mut i = 0;
        functions.push(Box::new(move || println!("Hello {i}")) as Box<dyn Fn()>);
        i = 1;
        functions.push(Box::new(move || println!("Hello {i}")));
        i = 2;
        functions.push(Box::new(move || println!("Hello {i}")));
        
        for func in functions {
            func();
        }
    }
But the most equivalent code would be the following which doesn't compile:

    fn main() {
        let mut functions = vec![];
    
        for i in 0..10 {
            functions.push(Box::new(|| println!("Hello {i}")))
        }
        
        for func in functions {
            func();
        }
    }
Re:Go: Redefining For Loop Variable Semantics by littlestymaar on Tuesday October 11, 2022 @04:55PM (#33169128)
That's the point: the misleading code doesn't compile because the `i` variable scope in limited to the current execution of the loop (and that's why you can't borrow it for `functions`'s lifetime), which is exactly how ES6 fixed this in JavaScript.
Re:Go: Redefining For Loop Variable Semantics by maxloh on Tuesday October 11, 2022 @02:24PM (#33167030)
It is a scoping problem IMO. In JavaScript variables are block-scoped, i will be out of scope after the loop.

It would be better if Python could support block-scoping too. (Maybe it is time for inventing a "strict mode" for Python)

Re:Go: Redefining For Loop Variable Semantics by kjeetgill on Tuesday October 11, 2022 @03:37PM (#33168190)
I've always referenced this as one of the subtle points of genius for how Java did lambdas:

by requiring captured variables to be final it removes a lot of ambiguity around what a variable name refers to. If you do want crosstalk between the inner and outer scopes you have to be more explicit and introduce a reference to talk through.

I love python, but I basically avoid this construction and use a single element list if I need it. I can never remember exactly how it works.

Re:Go: Redefining For Loop Variable Semantics by tomtung on Tuesday October 11, 2022 @08:55PM (#33171340)
Ah I have "fond" memories about once spending a good week debugging this, because I was also spawning threads/sub-processes in the loop, making this a Heisenbug that caused the program to crash every hour or so...