News for hackers. Stuff that matters.
Re:Go: Redefining For Loop Variable Semantics by jstimpfle on Tuesday October 11, 2022 @03:27AM (#33160386)
WTF. I've never been interested in Go after looking at it briefly a decade ago. This post brings back the weird feeling about it that I had then. Seems like taking the address of a local variable (item := item; append(..., &item);).

Variations of such perceived weirdness exist in many other languages with complicated "object models" as well, admittedly. Delphi has some strange stuff going on as well. Python has this weird "default list" thing going on. Most object languages don't let you even take the address of something (like the Go example shows) in the first place, but have only object references which I find unergonomic.

Re:Go: Redefining For Loop Variable Semantics by silon42 on Tuesday October 11, 2022 @03:42AM (#33160459)
I also feel there is a problem in that line, not necessarily just a for loop.

Maybe that's my C/C++/Rust experience.

Re:Go: Redefining For Loop Variable Semantics by tialaramex on Tuesday October 11, 2022 @04:36AM (#33160808)
In Rust we know we only gave the append function an immutable reference which lives until the next iteration. If append is OK with that, we're golden. Maybe internally it clones these references -shrug.

If append() needs the thing, not just an immutable reference which expires soon, its signature would demand we move one into it, and we don't have one so we'd need to e.g. make one with Clone.

Re:Go: Redefining For Loop Variable Semantics by maccard on Tuesday October 11, 2022 @05:10AM (#33161030)
> Maybe internally it clones these references -shrug.

That's what go does under the hood in 95% of cases. If I do:

func foo(a int) int { return &a }

func main() { fmt.Println(foo(3)) }

It does the right thing. It's just that in this case (and in lambdas) they got it wrong.

Re:Go: Redefining For Loop Variable Semantics by masklinn on Tuesday October 11, 2022 @05:12AM (#33161050)
TBF Rust also intentionally used the semantics Go is migrating to from the start, because:

- experience with the issue in languages with wider scoping e.g. it’s a common issue in JS, as well as Python (though slightly less so) - Rust’s iterators were originally internal so that was pretty natural

(1) is also why for(let and for(const have different scoping than for(var in JS: `var` has function scoping, `let` and `const` were introduced with block scoping, and for loops they were specifically specced with “inner” (per-iteration) scope.

Re:Go: Redefining For Loop Variable Semantics by Blikkentrekker on Tuesday October 11, 2022 @07:15AM (#33161908)
Aside from Python and the Shell, it never occurred to me that any language could possibly think of other semantics. It's news to me that C++ also assigns rather than initializes on each iteration.

It's simply a very bad idea that provides no use yet creates many bugs.

Re:Go: Redefining For Loop Variable Semantics by masklinn on Tuesday October 11, 2022 @07:53AM (#33162210)
The thing is it only create many bugs when you throw closures into the mix. So historically languages did it because it was easy to implement (create a counter, increment the counter, run the loop body).

Before the early aughts, closures were mostly really common in functional languages which tend towards immutable bindings (and immutability in general), and very closure-focused languages closured everything so didn’t hit that issue (e.g. you wouldn’t hit it in Smalltalk because your counter would be a parameter to a block, so closing over that was no issue).

It’s really in the 00s with the explosion of callbacks-pile-javascript (and more generally the functionalisation of imperative languages) that the problem became a serious concern: you loop over a thing, you start some sort of async operation (network request for instance), and you find out that despite the request being correct the entire thing goes wonky (then again things commonly went wonky which didn’t help).

Re:Go: Redefining For Loop Variable Semantics by Blikkentrekker on Tuesday October 11, 2022 @09:51AM (#33163361)
> The thing is it only create many bugs when you throw closures into the mix. So historically languages did it because it was easy to implement (create a counter, increment the counter, run the loop body).

It's a problem with references in general as this shows.

I also don't feel it's easier to implement at all.

One can either rewrite:

   for $id:var in $exp:iter { $code:body }
to:

   { let $id:var;
     while(True) {
       let result = $exp:iter.next();
       if(result.is_none()) break;
       $id:Var <- result.extract();
       $code:body
     }
   }
Or

   while(True) {
     let result = $exp:iter.next();
     if(result.is_none()) break;
     let $id:var = result.extract();
     $code:body
   }
The latter implementation is as far as I see easier, not more complex. Obviously all the code to create scoping already exists in the compiler and for-loops over an iterator work with a syntactic rewrite to an infinite loop with a break.
Re:Go: Redefining For Loop Variable Semantics by masklinn on Tuesday October 11, 2022 @11:16AM (#33164352)
> It's a problem with references in general as this shows.

Most languages don't have references, and in those that do before the issue was understood, the explicitness made it a much smaller issue.

> I also don't feel it's easier to implement at all.

> One can either rewrite:

Now try lowering to bytecode or assembly instead of high-level pseudocode.

Re:Go: Redefining For Loop Variable Semantics by Blikkentrekker on Tuesday October 11, 2022 @12:43PM (#33165548)
> Most languages don't have references, and in those that do before the issue was understood, the explicitness made it a much smaller issue.

But Go and C++ do, where this issue arose with or without closures.

> Now try lowering to bytecode or assembly instead of high-level pseudocode.

It doesn't matter, because as I said, all that is already in the compiler.

It would be needlessly complex and error-prone for compilers to hardcode custom code generation for such abstractions; it's transformed to something else the compiler already understands at a far higher level. I know for a fact that in Rust, for-loops already desugar to a simple infinite loop construct with a break at the H.I.R. level and all further optimizations only happen from there.

Re:Go: Redefining For Loop Variable Semantics by tialaramex on Wednesday October 12, 2022 @04:43AM (#33174430)
Yup. C# likewise described the change in semantics as just a different de-sugaring of their for each loop when that happened.
Re:Go: Redefining For Loop Variable Semantics by Blikkentrekker on Wednesday October 12, 2022 @06:16AM (#33174930)
Which makes it more confusing why it was originally as it was because it's really not harder to implement as any compiler implements it as desugaring before optimizations een occur and this form is simpler.

The only explanation I see is that they really gave it no thought at all whatsoever and it wasn't a tradeoff but simply not thinking clearly.

Re:Go: Redefining For Loop Variable Semantics by tialaramex on Tuesday October 11, 2022 @07:31AM (#33162041)
If you take a Rust for loop, de-sugar it to produce the actual loop { } which would run, and then modify that loop to have the Go semantics - with a single long-lived variable which is re-assigned each iteration, Rust detects the faulty Go cases of course.

[Edited: I tried to explain what's going on here, but I don't think my explanation was helpful so I've just left the surface]

Re:Go: Redefining For Loop Variable Semantics by masklinn on Tuesday October 11, 2022 @07:45AM (#33162150)
> Rust detects the faulty Go cases of course

Yes Rust’s ownership rules make it rather complicated to reproduce the faulty behaviour, as it’s about sharing mutable state which Rust intensely dislikes. You’d need to wilfully share (and update) internally mutable structures (cells, atomics) which is pretty noticeable and not something you do by mistake.

Re:Go: Redefining For Loop Variable Semantics by benhoyt on Tuesday October 11, 2022 @04:05AM (#33160608)
Yes, taking the address of a local variable is exactly what it's doing. But Go's compiler (and garbage collector) ensures it's safe to do that. The compiler will allocate on the stack where possible, and the heap where necessary, invisible to you. This is all very normal in Go.

I'm not sure prefixing your comment with "WTF" and your 10-years-ago dismissal helps the discussion here. Yes, as we've learned, this was probably the wrong decision, but it's not hard to see why it was done that way originally (C# made the same decision), and now they're having a reasonable technical discussion to try to solve it. And -- even though I've been bitten by this several times myself -- it's not a terribly common occurrence.

Re:Go: Redefining For Loop Variable Semantics by jstimpfle on Tuesday October 11, 2022 @04:10AM (#33160636)
I apologize for being snarky. "WTF" is an expression of surprise though, much more than it is criticism. My point was that these object models achieve a desired level of user-friendliness and safety at the cost of being less orthogonal and less composable (as compared to, say, C) and having weird corner cases and surprises that catch you off-guard.
Re:Go: Redefining For Loop Variable Semantics by jerf on Tuesday October 11, 2022 @01:06PM (#33165858)
It can be argued, based on the number of people who at some point write code in C that takes the address of a stack variable and returns it back out of the function scope, that the "less orthogonal" corner case that catches you off guard is the way C forbids that action. Do not mistake internalized concepts from a particular language as some sort of divinely approved dictate of how programming must work.

This was basically Dijkstra's point in his BASIC considered harmful post... I think in 2022 it should be C considered harmful for the same reason. C is not the base truth of computation. It isn't even very good. A language smart enough to analyze taking pointers and notice it can't put something on a stack and simply take care of it is, in my opinion, the one that is not catching you off guard... specifically, the "guard" that one must take in C around what is stack versus heap.

Re:Go: Redefining For Loop Variable Semantics by quietbritishjim on Tuesday October 11, 2022 @04:50AM (#33160901)
I can see why it's a surprise. Most languages that I know of fall into one of two types: (1) garbage collected and assigning a variable or passing to a function actually passes a reference to the object (certainly true of Python, Java, C#); (2) memory is manually managed and you can take the address of an object (C, C++, Rust).

That history makes it feel like "taking the address" is a really trivial operation - returning a numerical value that the compiler had access to at that point anyway. Here it's adding a reference to the object in some sense, and maybe even changing how it's allocated earlier in its lifetime (on the heap rather than the stack). I don't use Go and I agree that using &x for that operation feels a bit wrong as an outsider.

Re:Go: Redefining For Loop Variable Semantics by masklinn on Tuesday October 11, 2022 @05:46AM (#33161288)
Fwiw the issue occurs in langages of category (1) though only in a subset of the cases. Generally closures as they implicitly take references on their lexical context.

It also occurs in langages of category (2), specifically C++ lambdas where i think it can cause UAF/UB. I assume it also happens in C with the block extension (is that still Apple specific?) though I don’t know the details of that thing so maybe not.

Re:Go: Redefining For Loop Variable Semantics by quietbritishjim on Tuesday October 11, 2022 @06:48AM (#33161707)
The discussion here is specifically about using & to extend the lifetime. In the first case you mention, you don't use the & operator. In the second case, you do (at least with C++ lambdas), but there's no lifetime extension going on.
Re:Go: Redefining For Loop Variable Semantics by foldr on Tuesday October 11, 2022 @11:31AM (#33164559)
There's no special 'lifetime extension' operation involved. Semantically, everything is heap allocated† in Go. The garbage collector takes care of deallocating it when nothing references it anymore. In other words, at the level of the language semantics, all tracking of lifetimes is done dynamically at runtime by the garbage collector.

Go does in fact stack allocate variables which it can prove not to outlive their lexical scopes, but this is merely an optimization.

There's not really any such thing as a 'local variable' in Go. A variable has whatever scope it has, but there's nothing special, semantically speaking, about variables defined inside functions or inside loops.

>That history makes it feel like "taking the address" is a really trivial operation - returning a numerical value that the compiler had access to at that point anyway.

It is in fact a trivial operation in Go too, as I hope the above has clarified.

     ---
† Strictly speaking 'semantically heap allocated' is nonsense, but hopefully you know what I mean. There is no way to declare a variable in Go in such a way as to force it to be deallocated at the end of a particular lexical scope. A variable's lexical scope and its lifetime are entirely divorced (as is typical in a GCed language).
Re:Go: Redefining For Loop Variable Semantics by quietbritishjim on Wednesday October 12, 2022 @11:50AM (#33178765)
> There's no special 'lifetime extension' operation involved. ... The garbage collector takes care of deallocating it when nothing references it anymore.

I never used the word "special". As you say, adding a reference will mean the garbage collector won't deallocate it (until that reference is removed). In other words... its lifetime is extended. That's exactly what I meant.

> If the use of & in the example code is puzzling, it's probably because you're expecting Go to have some C-like concept of an automatic (i.e. stack allocated) variable ...

Not at all. In C++, you can use & on a reference variable and it will return the address of the object being referred to, regardless of whether it is allocated on the stack or the heap (or even statically allocated). Even in C, you can do &*x on a pointer to any object (which is silly by itself, but useful when combined with pointer arithmetic e.g. &x[3] translates to &*(x+3)).

> It [the & operator in Go] is in fact a trivial operation in Go too, as I hope the above has clarified.

Maybe I should have avoided the word "trivial" as its meaning is subjective, but I was careful to define what I meant by it: "returning a numerical value that the compiler had access to at that point anyway". Your comment just confirms that, as I said, it does more than that – it also adds a reference to the object.

---

To be clear, I'm not saying that it's bad or wrong that Go uses the & operator to mean this. Once you're familiar with the language, you probably get used to it very quickly. My point was just that it's a surprise initially if you're not familiar with the language, that's all.

Re:Go: Redefining For Loop Variable Semantics by foldr on Thursday October 13, 2022 @09:06AM (#33190445)
>Your comment just confirms that, as I said, it does more than that – it also adds a reference to the object.

It simply evaluates to the address of the object, just as it does in C. if you think the & operator is doing something in addition to this, I think that must just be based on a misunderstanding.

I am not quite sure what you mean by 'adding a reference' to the object.

Let's take this function:

    func foo() *int {
      var x int
      return &x
    }
All that happens is the following:

- An integer is allocated (and initialized to zero).

- The address of this integer is returned.

If we dig into the implementation, we'll see that the integer is allocated on the heap. As far as Go's language semantics are concerned, everything is allocated on the heap and left to the GC to clean up.

As an implementation detail, values that provably don't outlive their containing functions are (sometimes) stack allocated. As x outlives its containing function, it won't be stack allocated. That's it. There is no special operation of 'adding a reference' or 'extending a lifetime'. Nor does the compiler even analyze lifetimes except for the purposes of applying an optional optimisation which has no effect on the semantics of the program. If you turned this optimisation off (which you totally could) then there'd be no need for the compiler to worry about x's lifetime at all.

Re:Go: Redefining For Loop Variable Semantics by quietbritishjim on Thursday October 13, 2022 @12:31PM (#33193272)
> All that happens is the following:

> - An integer is allocated (and initialized to zero).

> - The address of this integer is returned.

That is not all that happens, at least down at the C/assembler level.

Let me illustrate what I mean. Consider this function, which also does both of these things (cobbled together from Google searches so please excuse incorrect syntax):

    func foo() uintptr{
      var x int
      return uintptr(unsafe.Pointer(&x))
    }
All that function does is allocate an integer (and initialise to zero) and return the address of that integer. Exactly the same as your function, right? Except it's obviously not - it doesn't extend the lifetime of the integer variable.

So why not? The GC somehow knows to ignore the number returned from my function, even though, under the hood, it's still stored in a register or stack location or whatever in exactly the same way as the address returned from your function. So how does the GC know to ignore it? Is that number somehow marked in a way that says "GC, when you're scanning memory looking for address-like numbers, don't pay attention to this one"? No. It doesn't look at the number in the first place because it hasn't been told to look at it.

In contrast, in your example, the memory address is not just returned from the function (in the C sense that it's put in a register for the caller to receive). It, additionally, somehow registers that memory address with the GC to let it know that there's another reference to that variable location. That is the extra thing that your function does that mine doesn't. And that magic happens (or at least starts) at the moment you use the & operator.

Re:Go: Redefining For Loop Variable Semantics by foldr on Friday October 14, 2022 @06:30AM (#33201711)
Yes, Go has a precise (i.e. non-conservative) garbage collector. It seems odd to me to think about that as some kind of special feature of the & operator. It's certainly not a surprising feature. Knowing that Go has a precise GC, one certainly expects the GC to know that the value of &x references x. If it didn't that would be a major bug.
Re:Go: Redefining For Loop Variable Semantics by vips7L on Tuesday October 11, 2022 @12:36PM (#33165452)
This cannot occur in Java. Any captured variables by lambdas must be effectively final.
Re:Go: Redefining For Loop Variable Semantics by origin_path on Tuesday October 11, 2022 @05:17AM (#33161086)
I think the bar to a "WTF" reaction here is lower because this is the sort of thing we've come to expect from Go. I'm in the same boat as jstimpfle where I looked at Go some years ago, found it to be remarkably quirky for a relatively traditionalist language that isn't trying to do lots of new ideas, and haven't yet regretted staying away.

It's not just this bizarre gotcha (the fact that C# had it too doesn't make it OK). It's that Go has so many of these cases where they took very strong positions on things and then later reversed their position only after many, many years:

- Generics

- Only one gc knob

- No backwards incompatible language changes

Also, Go has been around quite a long time now and we've all read quite a few rants about its surprising cases. How comes this one never came up before? The thread provides evidence that it bites people regularly. It suggests to outsiders that you can't easily evaluate Go by reading about it because there will be sharp edges that people aren't talking about simply due to the quantity of things that are even worse.

Re:Go: Redefining For Loop Variable Semantics by grey-area on Tuesday October 11, 2022 @05:23AM (#33161130)
You have remarkably strong opinions for someone who has not used the language much.

Personally after using it for 10 years I've been bitten very rarely by weird corners of the language and have enjoyed using it. My complaints are more around things I'd rather see removed (struct tags, panic, nils) and inconsistencies (built-in generics were quite limited, I quite like the design for generics they came up with though so I guess that is resolved once they update the stdlib).

Overall it's still my favourite language compared to others I'm forced to work in, I particularly like the decision to eschew inheritance.

Re:Go: Redefining For Loop Variable Semantics by doctor_eval on Tuesday October 11, 2022 @07:02AM (#33161803)
I agree with this take wholeheartedly. Go is a pragmatic language. Some of the design decisions make a lot more sense when you use it, and because Go seems to have a culture of utility and self reflection, I think you see more openness and constructive criticism than in some other languages I’ve used.
Re:Go: Redefining For Loop Variable Semantics by int_19h on Tuesday October 11, 2022 @05:17PM (#33169362)
C# made the same decision because it did not have closures initially, and so there was no practical way to observe the difference.

But Go did have closures initially, and worse yet, they already had C# as an example of how closures and loops interact. So they definitely had the opportunity to learn from that mistake, and I don't think it's unreasonable to ask why they did not.

Re:Go: Redefining For Loop Variable Semantics by Cthulhu_ on Tuesday October 11, 2022 @07:59AM (#33162281)
I mean Javascript does / did the same thing (var loop variables get hoisted to the function scope, so they are available outside of the loop; add to that that `range` creates a pointer and you have a perfect storm of weird, confusing things.

But this is an important point: They were aware of it; it was by design; they had the chance to change it before 1.0 and didn't, and now they are showing willingness to change. So many languages are resistant to change on the one hand (I mean Go is), yet not resistant to keep adding features (e.g. Java / JS).

Re:Go: Redefining For Loop Variable Semantics by udp on Tuesday October 11, 2022 @08:21AM (#33162465)
> Javascript does / did the same thing (var loop variables get hoisted to the function scope, so they are available outside of the loop; add to that that `range` creates a pointer and you have a perfect storm of weird, confusing things.

let is preferred nowadays in JS and doesn’t have the weird hoisting behaviour that var does/did. JS has neither “range” nor pointers though so I’m not sure what you mean by that.

Re:Go: Redefining For Loop Variable Semantics by avl999 on Tuesday October 11, 2022 @12:53PM (#33165687)
`&varName` is not equivalent to C version of the superficially similar looking operation.