News for hackers. Stuff that matters.
Re:Notes on structured concurrency, or: Go statement conside... by arghwhat on Wednesday April 25, 2018 @11:26AM (#16922377)
What.

This article proposes a "nursery", which is just a wrapped sync.WaitGroup/pthread_join/futures::future::join_all/a reactor that waits for all tasks to terminate/etc.

It then uses an exception-like model for error propagation to "solve" error handling (which is fairly easy to handle with a channel).

The construct is a decently usable, already applied tool to handle a set of problems, but the article takes the issue way out of proportions and overhypes the proprosed solution. The "with" example for benefits to not having a "go" statement seem rather bogus, especially seeing that such RAII constructs do not exist in Go (no destructors, remember?).

Trying to claim that "go" is as terrible as the original "goto" is ignorance of the original problems. Bad use of goto can be a nightmare to track (as the author tried to illustrate), but goroutines do not jump around, they branch from the main goroutine, following normal control flow from there. They are easy to follow, and the language is designed so that you can throw around with them and forget them without them causing you problems.

Also, this article is comparing a list of concurrency constructs and one parallism construct (pthread_create—threading.Thread doesn't count as parallism due to GIL) to callbacks, something which have nothing to do with concurrency at all. Very odd.

Re:Notes on structured concurrency, or: Go statement conside... by scott_s on Wednesday April 25, 2018 @11:32AM (#16922455)
I thought the article's comparison of go routines to goto was fair, in the context made: when calling a function in a language that allows go-routine like things (including pthread_create), one can't know if background tasks will be spawned. This is similar in the free-form time of unrestricted goto that one could not know if control flow really would return to the point lexically after the function call. His proposed abstraction does allow one to look after a nursery block and know whether or not background tasks are still running.
Re:Notes on structured concurrency, or: Go statement conside... by arghwhat on Wednesday April 25, 2018 @11:40AM (#16922552)
Knowing whether a background task has spawned is very different from not being able to follow the control flow (goto potentially jumping to an entirely different function body).

Now, while Go is designed mostly for you to not care about goroutines, there are some corner cases where one must know if a resource is used by anything, such as the chase of when you wish to close a file handle.

However, I'd argue that this is not related to go's concurrency model and the presence of background tasks at all. It's related to object lifetimes. This should be made clear at API surfaces.

A language solution to this would be Rust's lifetimes, not a new concurrency model.

Re:Notes on structured concurrency, or: Go statement conside... by marcus_holmes on Wednesday April 25, 2018 @12:02PM (#16922850)
But this concept does also solve that problem. Maybe more simply.

Rust is renowned for being a total headache with borrowing and lifetimes. Nurseries might be a simpler solution for this.

As you say, a corner case. But a common one...

Re:Notes on structured concurrency, or: Go statement conside... by lilyball on Wednesday April 25, 2018 @02:06PM (#16924300)
Nurseries and lifetimes are orthogonal concepts. In fact, Rust already has a "nursery", except it's generally called a scoped threadpool. They use lifetimes to ensure the threads can reference stuff on the stack without copying it. So a "nursery" is basically just a scoped threadpool with a 'static lifetime.
Re:Notes on structured concurrency, or: Go statement conside... by steveklabnik on Wednesday April 25, 2018 @03:11PM (#16924977)
There is some interesting stuff about this in the /r/programming thread https://www.reddit.com/r/programming/comments/8es8x3/notes_o...
Re:Notes on structured concurrency, or: Go statement conside... by stouset on Wednesday April 25, 2018 @10:21PM (#16928292)
I don’t think there’s any reason a nursery has to have a static lifetime.
Re:Notes on structured concurrency, or: Go statement conside... by eridius on Thursday April 26, 2018 @12:13AM (#16928743)
The description of the nursery is you can pass it around wherever you like. More generally, this concept seems to have been designed in a language without lifetimes. You obviously can have a scoped threadpool without a 'static lifetime, but if you want to pass it around to arbitrary locations then you do.
Re:Notes on structured concurrency, or: Go statement conside... by twic on Thursday April 26, 2018 @08:33AM (#16931232)
I think the whole point of the nursery's design is that you can create and destroy them during the life of your program, and that destroying them forms a barrier where the owner of the nursery waits for its children to finish. Having a nursery with a static lifetime in Rust would therefore be pointless, as it would live until the end of the program.
Re:Notes on structured concurrency, or: Go statement conside... by eridius on Thursday April 26, 2018 @11:50AM (#16932960)
I think you misunderstand. I'm talking about writing something like `ScopedThreadpool<'static>`, which means any external data referenced by the threadpool must have a static lifetime (or rather, it means the ScopedThreadpool cannot reference anything on the stack, because that would prevent you from e.g. returning it to your caller or passing it to another thread). The ScopedThreadpool itself can be created and destroyed whenever.
Re:Notes on structured concurrency, or: Go statement conside... by arghwhat on Thursday April 26, 2018 @10:15AM (#16932055)
I would definitely consider the "nursery" to be a newly created device (threadpool/reactor/...), where the resources used by the functions run in this "nursery" only has a lifetime requirement equal to the nursery.
Re:Notes on structured concurrency, or: Go statement conside... by arghwhat on Wednesday April 25, 2018 @04:17PM (#16925621)
It actually does not solve the problem of lifetimes. It only solves a very isolated instance of it.

For example, if we take the problem of an os.File, some library might store the reference, and use it unexpectedly in a later function call after you called Close. This presents the exact same issue as a goroutine holding it.

(I honestly have no other examples of this issue than files/connections being closed prematurely for Go.)

Re:Notes on structured concurrency, or: Go statement conside... by wybiral on Wednesday April 25, 2018 @08:01PM (#16927602)
I've never encountered a situation in Go where I'm passing file handles around like this.

In practice you'd probably have one goroutine manage the resource and its lifespan then other code would communicate with that goroutine using channels.

Re:Notes on structured concurrency, or: Go statement conside... by arghwhat on Thursday April 26, 2018 @03:48AM (#16929764)
You might pass an os.File or a network connection as an io.Reader or io.Writer to something that might keep it. The GC actually has an extension to sorta have destructors to help with leaking fd's from forgotten os.File's (runtime.SetFinalizer).

However, having coded Go for quite a few years by now, I haven't found it to be an issue at all.

Re:Notes on structured concurrency, or: Go statement conside... by noxecanexx on Wednesday April 25, 2018 @05:43PM (#16926490)
This is very true. The same way one doesn't know if a background was started is the same way you can't know if a file was written to or some global state was changed. I thought async/await solves most of his examples
Re:Notes on structured concurrency, or: Go statement conside... by wilun on Wednesday April 25, 2018 @01:20PM (#16923784)
One can also not "known" of all the other side effect that will happen.

Unless they are documented informally or encoded formally.

Re:Notes on structured concurrency, or: Go statement conside... by speleding on Wednesday April 25, 2018 @02:15PM (#16924391)
You realise that makes you sound exactly like an old Fortran programmer refusing to give up goto? A for loop is just a wrapped goto! ;-)

I think the author proposes a useful way to structure multi-threading. The comparison to goto isn't perfect and he needs to play fast and loose with some terms to keep the analogy working but he makes a good point.

I'm not convinced yet that the nursery pattern should be the only allowed way to start a thread but saying "you should use it unless you have a good reason not to" is a good provocation to get the discussion going.

Re:Notes on structured concurrency, or: Go statement conside... by platinumrad on Wednesday April 25, 2018 @03:10PM (#16924973)
"I think x is like goto."

"I disagree; I don't think x is like goto."

"Now that we've agreed that x is like goto, not wanting to give up x makes you like a big dinosauric dummy~"

Re:Notes on structured concurrency, or: Go statement conside... by kybernetikos on Wednesday April 25, 2018 @05:03PM (#16926144)
> 'A for loop is just a wrapped goto!'

corresponds fairly closely with

> a "nursery", which is just a wrapped sync.WaitGroup/pthread_join/futures::future::join_all/a reactor that waits for all tasks to terminate/etc.

The fact that the 'nursery' can and has been implemented with more 'primitive' constructs is actually a point of similarity with goto/structured programming not a point of difference. The GP didn't seem to spot this, despite making it their first complaint, so it was good that the post you're responding to did.

The post you're responding to is not calling the GP a dummy because of a disagreement, it's pointing out the apparent criticism the GP levelled was exactly the same criticism levelled against structured programming, and ultimately goto lost.

Re:Notes on structured concurrency, or: Go statement conside... by arghwhat on Wednesday April 25, 2018 @05:20PM (#16926294)
As someone whose work includes hardware driver programming, I'd rather not give up goto. It would make a lot of error handling extremely cumbersome, and much less readable. :(

However, I find "go" and "goto" to not intersect at all. I've written this many times in other comments on this thread, so I'd rather not type it out again, but the TL;DR: is that "goto" can make understanding a function when read difficult, while "go" is clear when read. No function is understood if never read, and spawning "background tasks" is a core part of asynchronous programming (and thus not an unusual side-effect).

The "nursery pattern" is a decent construct that I have used quite often whenever I felt a need, but it doesn't appear to really solve any issues mentioned in the post. I also elaborated on this quite a few times already, so TL;DR: the only real problem of goroutines is things like references to potentially closed objects, but any method may end up storing an internal reference used at a later call, making the issue not related to concurrency.

Re:Notes on structured concurrency, or: Go statement conside... by stouset on Wednesday April 25, 2018 @10:30PM (#16928321)
> while “go” is clear when read…

I believe our industry’s several decades of demonstrated inability to write concurrent code correctly disagrees with you.

Languages like go (and Rust, to a greater degree) have improved the situation. But there are still a ton of sharp edges that existing approaches still have, and—often—they’re ones that aren’t readily apparent until after a project starts growing and begins unearthing issues that are rare, only happen on faster hardware, or require higher levels of concurrency than we’re previously used.

Re:Notes on structured concurrency, or: Go statement conside... by wilun on Thursday April 26, 2018 @02:39PM (#16934400)
> I believe our industry’s several decades of demonstrated inability to write concurrent code correctly disagrees with you.

But mostly so for the reasons nurseries try to solve (and actually manage to somehow solve some)? I'm not convinced.

Lifetime of threads is usually easy compared to races/deadlock/etc. on the resources they use.

Not to say we shall not use nurseries where applicable. But is it the new fundamental structure concurrency should be based on? Debatable.

My wild guess is that the Rust approach of concurrency (even if it is not exactly on the same subject, but we are trying to find the fundamental way of structuring things) will have more impact.

Re:Notes on structured concurrency, or: Go statement conside... by arghwhat on Thursday April 26, 2018 @04:01AM (#16929859)
> I believe our industry’s several decades of demonstrated inability to write concurrent code correctly disagrees with you.

Not at all. Concurrent/parallel code isn't particularly difficult to write—there's just a unique class of problems related to it that might occur, but that does not mean it is hard.

Rather, the industry has through several decades demonstrated an inability to write bug free code in general. I wouldn't blame concurrency/parallelism for this.

Re:Notes on structured concurrency, or: Go statement conside... by camgunz on Thursday April 26, 2018 @10:42AM (#16932253)
I guess we can debate what "hard" means, but I think "adds a lot more problems that are much harder to reason about and avoid" is exactly the definition of hard. If it's not, I'm honestly not sure what is. Torn reads, deadlocks, concurrent modification, the effects on optimizers, the overhead of scheduling and lock management. Each of these are deep, complicated issues you only have to deal with when writing concurrent (well, parallel) code.

But yeah I mean, if you mean it's pretty easy to call pthread_create then OK. But if you at all care about your program working, then parallel code is much, much harder to write.

Re:Notes on structured concurrency, or: Go statement conside... by arghwhat on Thursday April 26, 2018 @11:52AM (#16932975)
All the problems you present go into the category of sharing state. This is the unique problem presented by concurrent programming, which is no harder than all the other problems in programming. This does not mean that concurrent programming is hard, but that just like all other programming, it has some sharp edges.

The issues related to sharing state can be easily avoided by, for example, using a CSP-style paradigm (channels in Go). In Go, this only leaves behind a deadlock, which are automatically panic, making it incredibly easy to debug.

Overhead of scheduling and effects on optimizers are not related to concurrency/parallelism. The scheduler is affected by many things even under single-threaded execution which is far beyond the scope of normal application developers, and optimizers are largely unaffected by concurrent/parallel programming (although shared structures must internally present memory barriers). The optimizer is also beyond the scope of normal application development.

If you are doing development that requires precise control over optimizations and scheduling (like I do), then all bets are off.

However, with "normal" (i.e. not-processing-8x100Gb/s streams) programming, concurrent programming is a breeze unless you intentionally shoot yourself in the foot.

Re:Notes on structured concurrency, or: Go statement conside... by camgunz on Friday April 27, 2018 @11:52AM (#16941703)
> ...which is no harder than all the other problems in programming.

Well but for almost all of those problems you have something to help you. Yes manual memory management is hard, thanks GC/lifetimes. Yes error handling is hard, thanks exceptions (kind of). The point of OP is that practically all of the things we've built to help us with parallel/concurrent programming are still very low level, and your program still has to deal with the fallout in a way that it doesn't with GC or exceptions or other helpful abstractions.

> The issues related to sharing state can be easily avoided by, for example, using a CSP-style paradigm (channels in Go). In Go, this only leaves behind a deadlock, which are automatically panic, making it incredibly easy to debug.

There are only a few languages/platforms where CSP is feasible, and very few (none?) of them achieve the performance of CSP in Go because they don't/can't use segmented stacks. And even when using CSP you still need synchronization primitives. Parallel Go code uses mutexes everywhere. CSP in Go helps with _implementing_ parallel programs, but the Go team provides a lot of extra tooling to help _debug_ parallel programs that's well outside the definition of CSP --> the deadlock panic you cite is a good example actually.

> Overhead of scheduling and effects on optimizers are not related to concurrency/parallelism.

I disagree; mostly my evidence is "just google for volatile in C".

~~~

In general I still think parallel program is in a separate class of problems, but even if we stipulate it's just as hard as other problems, I still think we have much better tools for dealing with the other problems. I think we're getting there -- and CSP is part of that -- but it's definitely not the case that we've settled on a solution for 99% of problems.

Re:Notes on structured concurrency, or: Go statement conside... by arghwhat on Friday April 27, 2018 @12:39PM (#16942146)
> ... and your program still has to deal with the fallout in a way that it doesn't with GC or exceptions or other helpful abstractions.

[citation needed]. What in the world is this "fallout"?

You can of course chose not to use CSP or some other abstraction, but that would be like chosing to do manual memory management (which you can also do in Go through cgo if you'd like), and the "fallout" is identical: You are on your own. CSP is of course not magic, but neither is a GC or lifetimes. You must always know the tools you are working with, as they have their own issues that must be dealt with.

If we look at the concerns in the article, Rust's lifetime actually remove those issues entirely, much in line with its "fearless concurrency" motto. Go's CSP is opt-in-although-aggressively-recommend, and is by no means a low-level construct.

A compare-and-swap is a "low-level" construct. You can of course pull in atomic primitives if you feel like doing so, but then you are choosing to go low-level.

Also, exceptions are a terrible, terrible thing. They make error handling much, much worse.

> There are only a few languages/platforms where CSP is feasible, and very few (none?) of them achieve the performance of CSP in Go because they don't/can't use segmented stacks

I do not see how this claim makes sense. Segmented stacks are an unnecessary green thread implementation detail that can both harm and benefit performance (if you use FFI, they harm performance a lot), and is neither necessary for green threads, nor related to CSP at all.

CSP works just fine in other languages. You can fully execute a CSP paradigm in C, just like you can avoid it altogether in Go and Rust.

Although, while you can avoid CSP in Rust, you still can't create the issues presented in the articles due to lifetimes. You would have to quite explicitly and intentionally get your hands dirty with unsafe code in order to shoot yourself in the foot here.

> I disagree; mostly my evidence is "just google for volatile in C".

"volatile" has nothing to do with anything here. Not only that, it is an entirely useless construct, as it does not provide reordering guarantees (that is, while subsequent reads see a previous write, the reads and writes may be reordered so that the reads now happen before the write). Proper constructs use memory barriers.

However, neither of these constructs have bad effects on optimization, unless you are aiming for code that executes fast without working at all. Also, memory barriers are low-level primitives. Unless you are designing synchronization primitives, you shouldn't touch them.

"volatile" and memory barriers have no effect on process scheduling at all. It does, however, necessarily affect the CPU instruction pipeline, but once you get your hands this dirty, you're way out of the scope of high-level programming safety net. Down here, we work with assembly.

Re:Notes on structured concurrency, or: Go statement conside... by sterlind on Thursday April 26, 2018 @07:54PM (#16937167)
Interesting; I program drivers but avoid goto entirely:

  NTSTATUS err = S_OK;
  if(!FAILED(err)) err = Op1();
  if(!FAILED(err)) err = Op2();
  ...
  if(FAILED(err)) /*cleanup*/
Re:Notes on structured concurrency, or: Go statement conside... by arghwhat on Saturday April 28, 2018 @12:39PM (#16948402)
That would work, but I'm not sure I find it that pretty. Plus, it assumes all "ops" have compatible error output (which is unfortunately rarely the case).

What I instead do is usually:

    int some_func() {
        int err = OK;
        if (some_op() != SOME_OK_CRITERIA) {
            err = SOME_OP_FAILED;
            goto error;
        }
        ....


    error:
         // do necessary cleanup

         return err;
    }
I'm also mostly responsible for Linux and BSD kernel mode driver, as well as our user-mode driver (most of our driver exist as a shared user-mode component)—I wipe the Windows kernel mode driver off on someone else. :)

I have no allergy to goto at all, as long as its not used in absurd fashions (like how it's used in a duffs device). If a goto seems easier/prettier, I use a goto.

Re:Notes on structured concurrency, or: Go statement conside... by justinpombrio on Wednesday April 25, 2018 @04:03PM (#16925491)
> The "with" example for benefits to not having a "go" statement seem rather bogus, especially seeing that such RAII constructs do not exist in Go (no destructors, remember?).

You've greatly underestimated how general this problem is.

First, Python's `with` statement has nothing to do with destructors. From the PEPM for `with`:

  with VAR = EXPR:
      BLOCK
  
  which roughly translates into this:

  VAR = EXPR
  VAR.__enter__()
  try:
      BLOCK
  finally:
      VAR.__exit__()
`exit` is a method, not a destructor. As a result, Go could easily gain a `with`-like construct.

Moreso, the issue applies even if there's nothing like `with` in the language. If I'm reading a function definition, and that definition uses the "acquire, try, finally, release" pattern that is the desugaring of `with`, then it sure would be nice to know that nothing from BLOCK is running after the `finally` statement has run. `go` breaks that assumption.

Re:Notes on structured concurrency, or: Go statement conside... by arghwhat on Wednesday April 25, 2018 @04:15PM (#16925608)
I mentioned RAII because the text around "with" mentions its use with RAII.

Go does not have exceptions (panic is not meant as "normal" flow control), and therefore has no use for a "with" construct. "defer" is used for a somewhat similar purpose. Go does not have destructors, and therefore has no possible implementation of RAII.

However, none of this applies to goroutines. A goroutine only gives errors if the author decides that such is necessary. If so, it will likely be through an error channel. There is no unexpected code paths through such readout, rendering "with" and RAII useless.

So again, for a post that was very focused on complaining directly about the "go" keyword, I was expecting something applicable to Go.

Re:Notes on structured concurrency, or: Go statement conside... by zamalek on Wednesday April 25, 2018 @02:44PM (#16924690)
Agree with everything. I think the misunderstanding here is that goroutines are not classical threads (which would share many problems with `goto`) - they exist somewhere between coroutines and actors (because of channels+select). Actors are an established and mature solution to many headaches surrounding concurrency and parallelism.

Goroutines share more in common with method calls (which are a form of branching) and even with single-threaded scenarios, methods can do surprising things - especially if you have shared global state.

Additionally, if you control how data is shared (such as message passing - channels) you shouldn't have to be concerned about what the other thread is doing - so long as it reacts to messages that you send to it.

Re:Notes on structured concurrency, or: Go statement conside... by tomtheelder on Wednesday April 25, 2018 @03:22PM (#16925098)
I don't think that's a misunderstanding at all. The argument is all about control flow and programmer understanding. The actual underlying mechanisms aren't what he is arguing against, instead he dislikes the potential for programmer confusion when code is being executed that wasn't expected. That is 100% possible with goroutines.

Method calls aren't really branching. The code path is still totally linear. You could basically copy paste the code in the method definition in place of its call and get the same outcome (not literally but you know what I mean). For goroutines this is not the case.

I think your points about state ownership are right on, but that's kind of the author's whole point. Right now, there are a lot of things you have to be very conscious of to write good code that executes cleanly using goroutines/threads/etc. That is very much the same as writing good code with gotos. It's 100%, unequivocally possible to write good code using gotos (every control flow structure dijkstra proposed can be represented with them), it just requires a lot of added thought, and the potential for mistakes is much higher.

The author is not proposing any functionality that doesn't already exist, just a new control pattern to reduce the chance of creating problems.

Re:Notes on structured concurrency, or: Go statement conside... by arghwhat on Wednesday April 25, 2018 @04:30PM (#16925782)
> That is 100% possible with goroutines.

It really isn't. The big difference is that "goto" can lead to code you read being grossly misunderstood due to complicated flows, potentially even absurd things like jumping to a different function. (And as you mention, good code can be written with goto's—it can make particularly error handlers much easier to read in C.)

This is not the case at all with "go", which is extremely clear as to what it does and how the flow will go.

However, as a caller, you do not know the flow of the function you call unless you read it. As with any type of asynchronous programming, a function might have scheduled something for later execution: It might be a future, a promise, a timeout/interval, network receive callbacks, or a goroutine.

That is, strictly speaking, none of your concern as long as the function lives up to its contract. If not, all bets are off regardless.

> ... it just requires a lot of added thought, and the potential for mistakes is much higher.

I get your point, but I must disagree that asynchronous programming (which is what goroutines is merely an implementation of) increase the complexity of writing good code, nor increase the likelyhood of state ownership mismanagement in any measurable fashion.

> The author is not proposing any functionality that doesn't already exist, just a new control pattern to reduce the chance of creating problems.

I don't really find that the author suggests anything new at all. Rather, the author takes an existing blob of code and claims a new benefit from it.

EDIT: erroneous "in which case" replaced with "if not", as initially intended.

Re:Notes on structured concurrency, or: Go statement conside... by wpietri on Wednesday April 25, 2018 @09:03PM (#16927968)
> I must disagree that asynchronous programming increase the complexity of writing good code

Could you say more about why? It seems to me that purely synchronous code is much easier to reason about than async code, in that a) less is happening at once, and b) things happen linearly. That seems less complex almost by definition.

I'd certainly believe that things like goroutines are better than the threading that came before, with way less cognitive load. But I can't see how it's less complex than linear code.

Re:Notes on structured concurrency, or: Go statement conside... by arghwhat on Thursday April 26, 2018 @10:39AM (#16932230)
> But I can't see how it's less complex than linear code.

I am being misinterpreted here (in a peculiar way that has happened before on HN...). I wrote that it does not increase the complexity "in any measurable fashion".

It seems that you interpreted this as "concurrency is less complex than non-concurrency", when in reality I am saying "concurrency is a tiny bit more complex than non-concurrency" (extra emphasis on "tiny").

Now, back to the topic:

Asynchronous programming is strictly speaking the simple idea that code will be run later when some conditions are met. For example, running a callback when a network response is received from the "fetch" API in modern JS, with all I/O handled behind the scenes by an event engine of sorts.

I do not believe that this concept provides any measurable increase in cognitive load. You only concern yourself with these devices where they are used, and they are extremely simple to wrap your head around.

In another model, you may have full parallelism and shared memory access, in which case you need to be more careful to only use thread safe structures. Even then, unless you do something stupid™, everything is fine, and the cognitive load is increased globally but only mildly so with a model similar to goroutines and channels.

Finally, you may have to design and use synchronized/atomic data structures, in which case things increase in difficulty. While this does increase cognitive load in a small area of the code, I does not increase the cognitive load over the entire application. However, just like not all application need to design a cryptographic protocol, not all applications need to design concurrent data structures even if it uses concurrency. Thus, one should additionally note that this localized overhead is not a universal overhead of all concurrent programming.

BTW, I do not find that goroutines present less cognitive load than pthreads ("threading") and a CSP library. They simply present syntactic sugar for a better developer experience. They also give an M:N threading implementation unlike pthreads, but that's an implementation detail.

Re:Notes on structured concurrency, or: Go statement conside... by wpietri on Thursday April 26, 2018 @04:58PM (#16935691)
Sorry, but I'm just not seeing it.

Having a magic, behind-the-scenes event engine with a queue containing an unknown number of items executing in unpredictable order seems definitionally way more complex than not having any of those things and running the code in strict linear order. Performance is way better, sure, but at a cost of complexity and cognitive load.

Sure, the difference is tiny if everything works smoothly. But if everything worked smoothly, we wouldn't have jobs.

Re:Notes on structured concurrency, or: Go statement conside... by arghwhat on Saturday April 28, 2018 @12:29PM (#16948361)
This is a bogus argument: "abstractions are complexities and lack control".

This argument can be equally applied to any other abstraction or convenience functionality, such as garbage collectors (unpredictable destructor execution, resource consumption and performance), OS schedulers (unknown number of items executing fighting for time quanta, unpredictable performance, unpredictable latency), high-level languages (unpredictable code generation, lack of ability to express proper intention related to processor abilities), etc.

However, the problem here is that your expectation of the abstraction is wrong, not that they introduce complexities. Furthermore, all abstractions give up control to the abstractions so that you do not have to concern yourself with it. You cannot both have full control over functionality and not concern yourself with the functionality.

An even engine is a scheduler like the OS scheduler. It is designed to take care of scheduling, making it not a concern for you, hiding the implementation (effectively making it "unpredictable"). If you want control, you have to give up the abstraction and go straight on the iron, just like you would have if you want precise memory behavior.

However, considering this a "complexity" would be wrong. There is nothing complex about asynchronous programming's lack of predictable execution order. The contract is that the code will execute when the event arrives, nothing else. This is by no means a complexity. In many engines, the execution of your code isn't even concurrent, only having I/O run in the background.

(I am intentionally ignoring buggy event engines, just like we ignore buggy GC's, kernels, compilers, CPU's, etc.)

Re:Notes on structured concurrency, or: Go statement conside... by dbaupp on Wednesday April 25, 2018 @09:47PM (#16928173)
> I think the misunderstanding here is that goroutines are not classical threads (which would share many problems with `goto`) - they exist somewhere between coroutines and actors (because of channels+select).

I don't think that comparison makes sense. Channels and select are not a property of goroutines themselves, nor is using those the only way to communicate with other goroutines. The actual functionality of a goroutine is very similar to a thread, even if the broader language/conventions push it towards a coroutine/actor.

> Goroutines share more in common with method calls (which are a form of branching) and even with single-threaded scenarios, methods can do surprising things - especially if you have shared global state.

But they can't run in parallel, or data race. Goroutines, like classical threads, can have data races.

> Additionally, if you control how data is shared (such as message passing - channels) you shouldn't have to be concerned about what the other thread is doing - so long as it reacts to messages that you send to it.

This is also true of threads. Message passing is a layer on top of, and agnostic to, some concurrency mechanism. Go might make it slightly syntactically simpler than languages that use true OS threads (much simpler than C's pthread_spawn, but not particularly different to Rust's thread::spawn(some_closure)), but that's a syntax layer.

Re:Notes on structured concurrency, or: Go statement conside... by cryptonector on Wednesday April 25, 2018 @10:53PM (#16928444)
Callbacks have plenty to do with concurrency: first you compile everything into CPS (where callbacks are continuations) and then a trivial event loop nets you cooperative multitasking. When you don't have language support for this but you need concurrency, you just write something resembling CPS by hand and call it "callback hell".

> (pthread_create—threading.Thread doesn't count as parallism due to GIL)

Eh? Why would you have a GIL? Maybe in Go, but so what, just don't use such a language/run-time.

Re:Notes on structured concurrency, or: Go statement conside... by ellyagg on Wednesday April 25, 2018 @06:05PM (#16926718)
Can I just mention that anyone who begins their comment with

    What.
...and:

    [whatever] is just a [whatever]
...comes across as a bit of an asshole?

Why not just make your point instead of investing a lot in making sure everyone knows that you're smart and the other person is dumb?

Maybe Knuth was just as rude to Dijkstra, who knows?

Re:Notes on structured concurrency, or: Go statement conside... by arghwhat on Thursday April 26, 2018 @03:51AM (#16929789)
Of course you can.

I felt that such "aggressive" countering was justified considering the equivalently grand claims of the post ("EXTREMELY_POPULAR_CONSTRUCT considered harmful").

Re:Notes on structured concurrency, or: Go statement conside... by fixermark on Wednesday April 25, 2018 @07:35PM (#16927433)
> It then uses an exception-like model for error propagation to "solve" error handling (which is fairly easy to handle with a channel).

If none of your code ever actually waits on the error channel when you spawn a goroutine, what happens?

Is there any circumstance where that behavior is preferable to guaranteeing something must wait on the error channel?

Re:Notes on structured concurrency, or: Go statement conside... by arghwhat on Thursday April 26, 2018 @03:55AM (#16929824)
Go's error paradigm is through return values that can be ignored, so an error channel would mimic "non-async" error handling in this case: Checking an error is is always up to the developer.

The discussion about error paradigms (exceptions vs. globals vs. error values, forced checking vs. free choice, etc.) is quite a big one, and arguably an entirely different topic.

Re:Notes on structured concurrency, or: Go statement conside... by tomtheelder on Wednesday April 25, 2018 @03:15PM (#16925022)
It really is a whole wrapped up group of existing constructs, but that doesn't mean there isn't merit in it. The author mentions how this was central to Dijkstra's proposal for structured programming: "And now that Dijkstra understood the problem, he was able to solve it. Here's his revolutionary proposal: we should stop thinking of if/loops/function calls as shorthands for goto, but rather as fundamental primitives in their own rights – and we should remove goto entirely from our languages."

> The "with" example for benefits to not having a "go" statement seem rather bogus, especially seeing that such RAII constructs do not exist in Go (no destructors, remember?).

I don't really understand what you mean by this. You would certainly need a different mechanism in Go than in Python, but I fail to see how that is a criticism of the argument he is making. He has built one implementation in one language, clearly it would look different in other languages.

> Trying to claim that "go" is as terrible as the original "goto" is ignorance of the original problems. Bad use of goto can be a nightmare to track (as the author tried to illustrate), but goroutines do not jump around, they branch from the main goroutine, following normal control flow from there. They are easy to follow, and the language is designed so that you can throw around with them and forget them without them causing you problems.

Why exactly is branching better? With "go" statements, you can easily end up with goroutines that you aren't even aware of floating around doing things that you aren't aware of. His example of calls in external libraries is probably the best illustration. Currently, you could call what you think is a simple function from a library, and end up with a whole host of goroutines you didn't expect floating around, doing things and using resources. Articles about writing good libraries [0] have to make points about how you need to take care of this stuff.

Gotos create problems when you end up surprised about what code is being executed, making tracing execution paths difficult. Branching creates a different set of problems: most crucially that you can have code being executed and not even realize it's happening. The issues are different, but his analogy is really spot on IMO. Both of these things can be solved with code reading and debugging, but the author's whole point is that with an improved abstraction you no longer have to worry about that sort of thing.

> Also, this article is comparing a list of concurrency constructs and one parallism construct (pthread_create—threading.Thread doesn't count as parallism due to GIL) to callbacks, something which have nothing to do with concurrency at all. Very odd.

These are all related concepts. They are about doing different things "at the same time," just with different definitions of what the phrase means. The problems the author are describing are considerably worse in a scenario where actual processes are being spawned, but the nursery pattern is relevant in all of them. It's about control flow, not capabilities. Even using threading with a GIL can create execution paths that you aren't really aware of until you actually look through the code or debug.

Re:Notes on structured concurrency, or: Go statement conside... by arghwhat on Wednesday April 25, 2018 @04:09PM (#16925546)
> but that doesn't mean there isn't merit in it

Of course not. It also isn't original, or a "silver bullet" that I find relevant to general concurrent programming.

> I don't really understand what you mean by this.

I was trying to keep my rant a bit short, but my point is that the only practical example for a post that puts a lot of effort into complaining about a core Go construct (it's the title) does not even remotely apply to Go.

This is partly due to Go being shaped around this very core construct.

> Why exactly is branching better?

"goto" can (sometimes, as there are many valid usecases) be problematic as it can make control flow very obscure when read. "go" is extremely clear to read, and only has the concern that you don't know if a given function has created goroutines. It is of course usually described by documentation, or implicit from functionality that such thing will occur, and unlike goto, the code is extremely clear about what is going on if you read it.

Furthermore, whether a function call has created goroutines is by itself not a concern. What can be a concern would be if some types of resources that must be closed (i.e. a file) is referenced after you close it. That is not related to concurrency, but lifetimes.

By lifetimes, I do not necessarily mean Rust-style compiler-enforced lifetimes (which I like), but simply API contracts. A library may store a reference to something you pass, and may use this reference in later calls, potentially after you invalidated the reference by closing a file descriptor.

A joined execution does not even remotely solve this problem, as it is not related to concurrency. It solves a different problem, related to just managing concurrent execution.

> These are all related concepts. They are about doing different things "at the same time, ...

They are not at all. Callbacks are often used together with certain types of concurrency constructs, such as an event-loop that can call callbacks upon various events. In JS, for example, the concurrency construct is a single global event-loop that calls tasks/microtasks.

Callbacks themselves, however, are not a concurrency construct. Thus, comparing them to concurrency constructs is very weird.

(A SAX parser is an example of a non-concurrent use of callbacks.)

> Even using threading with a GIL can create execution paths that you aren't really aware of until you actually look through the code or debug.

It does not create execution paths that are not easily visible (you know nothing about a program until you look through the code), but yes, threading.Thread can lead to some unanticipated behavior.

Note that I was excluding this as a parallelism construct, not a concurrency construct. The behavior will be some form of cooperative multi-tasking.

EDIT: too much "however".

Re:Notes on structured concurrency, or: Go statement conside... by tomtheelder on Wednesday April 25, 2018 @08:36PM (#16927820)
Ah I get what you mean about the use of go as an example.

Also, I think when the author refers to callbacks they are referring to callbacks in the JS sense, that is to say callbacks in the context of an event loop type environment. The author certainly wouldn't care about other types of callbacks, since they do not create the split execution paths that he is describing.

The distinction you are trying to draw between concurrency and parallelism, while totally valid, isn't particularly relevant here. The problems the author is describing are almost entirely to do with developers writing and understanding code, and they generally apply to both parallel and concurrent systems. The user experience of writing parallel and concurrent code can often feel extremely similar, so the tools the author is describing will tend to apply in both situations.

Re:Notes on structured concurrency, or: Go statement conside... by wybiral on Wednesday April 25, 2018 @03:49PM (#16925350)
Most articles riding on the "x considered harmful" meme are clickbait without much content.
Re:Notes on structured concurrency, or: Go statement conside... by stouset on Wednesday April 25, 2018 @10:33PM (#16928342)
Nothing could describe this post (or anything on the author’s site in general) less accurately than your comment.
Re:Notes on structured concurrency, or: Go statement conside... by wybiral on Thursday April 26, 2018 @12:07AM (#16928720)
How so?
Re:Notes on structured concurrency, or: Go statement conside... by scott_s on Thursday April 26, 2018 @10:43AM (#16932273)
Because the author engages directly with the core point Dijkstra made, and proposes a new abstraction which parallels with that core point. The author does so in a clear and thoughtful way which cannot be dismissed out-of-hand.
Re:Notes on structured concurrency, or: Go statement conside... by arghwhat on Saturday April 28, 2018 @11:56AM (#16948216)
And yet misses the point entirely, presenting non-original work that only acts as a band-aid on a symptom, rather than dealing with the cause (which is not related to concurrency constructs at all).