Take it back to the goto analogy: can you formally verify goto? Yes. Does that mean it's good to include as a language primitive? No.
You are basically taking an entire argument, ignoring its merits, and saying "but you can do it another way". You are exactly right, but you haven't rebutted the fact that structured concurrency is philosophically superior.
I love rust and the community; they will get to the truth of this argument eventually. But I suspect the truth is that "njs was right", and I hope the it's sooner rather than later.
> But I suspect the truth is that "njs was right", and I hope the it's sooner rather than later.
I'm not sure. The problem can and has been solved without the manually "pass the nursery object around" ""escape hatch"" for the general case of threads (because no: having the execution of spawning functions delayed by the lifetime of thread is arguably reasonable is some cases, but certainly not the general case of what threads are useful and used for)
It is still useful for tons of existing languages as a pattern anyway, but only if the use cases are suitable.
But the author is focused on a narrow use case of threads (and a narrow subset of the problems they introduce), and present their solution as a general truth and new fundamental control structure of computing, independent of already existing, in production, and arguably better solutions; and independent of analyzing the new problems their silver bullet introduces.
I agree that Rust has a great model that seems to lead to very solid code, but “formal verification” is a high bar to clear.
But I don't know how to call it, and it is way more checked on some aspects (well, obviously, we are not talking about checking each program against a spec...) that the competition. And by that, I mean that the competition is actually not even trying...
> Then our guarantee is lost: the operations that look like they're inside the with block might actually keep running after the with block ends, and then crash because the file gets closed while they're still using it. And again, you can't tell from local inspection; to know if this is happening you have to go read the source code to all the functions called inside the ... code.
But actually you can "tell", or even better have a type system good enough to prevent such mistakes entirely, and the author even knows a bit about Rust, yet fails to state that at least that item is a solved problem there (by using a different and arguably more general approach).
Now I don't know enough to decide whether something is missing on the panicking background thread front, but if it does that seems very solvable.
I don't buy that spawning threads (or even moral equivalents) and multiprogrammation is in a situation similar to unstructured use of goto anyway. You have tons of problems applicable to one and not the other. And of course resource management is hard to get right with threads. But we have at least a production example of a language that get it right on some points by leveraging more general ideas (and the other difficult points are mostly not addressed by the nursery idea, anyway).
However, I agree with you that there's no one-size-fits-all approach to concurrency. Sometimes you want long-lived tasks that communicate—the actor model, in other words—in which case fork/join doesn't buy you much. I do think that fork/join is frequently what you want, though.
I believe that python's `async for` construct (which may be stolen from c#, but I'm not sure) does this, and another user mentioned Promise.all which in javascript I believe allows the child promises to resolve concurrently.
In both cases, these are for concurrency, not parallelism.
I don't believe either of the constructs you've mentioned do this, and I'm not aware of any that do.
The trivial counter example would be a deeply nested `setTimeout(..., 5000)` inside the javascript code.
It doesn't matter if you've called Promise.all or not.
Also, regarding parallelism: Obviously async/await are for that; this is basically pitching an equivalent construct for parallel processing.
I think this is novel, frankly, and the alternatives being thrown around by people are by people who didn't read the article.
Well of course, setTimeout isn't async/await based, its callback based. If instead you only had this api, you could make that formal assertion:
await promiseSetTimeout(time).then(...);
So basically, async/await provides this as long as you only use async/await. In JS you can't formally assert this because there are functions that subvert the normal control flow, but if you outlaw those, you absolutely can.
Well, what if you don't?
That's the point.
The proposed construct does not require that: I'm not saying its better or worse; I'm just pointing out that it's different.
Obviously if you choose to avoid branching code in your functions, they won't branch; what's novel here is that the closing scope (ie. in this case, the collapse of the python with block) triggers a collection of all the ambient tasks started in that context.
I'm sure you could implement something similar in javascript, but it is not the same as Promise.all.
This is kind of a silly question. Its always possible to subvert a safe construct system if you try hard enough. You can write unsafe blocks in rust. You can pass in a callback to a nursery, as is described in the article.
>The proposed construct does not require that
Well, kind of. The article actually explicitly states that
>Here's a simpler primitive that would also satisfy our flow control diagram above. It takes a list of thunks, and runs them all concurrently:
which is equivalent to promise.all, satisfies the invariant nurseries do, except in very specific circumstances (unbounded while loop-y constructs). And you just simply can't use promise.all in that situation.
I don't agree that the proposed idea is fundamentally new and amazing; but I think it is novel, and there may be some value in being able to semantically bind tasks to execution points, specifically when the tasks are spawned in naive (or uncontrolled, eg. library) code, and have not explicitly opted in to the scheme.
Does this particular implementation do that perfectly? No, probably not.
...but I think the idea of it has some merit.
There's more value here than just waiting for a series of deferred tasks that you have explicitly created, and explicitly opted into a specific concurrency workflow with.
Note specifically these parts of the proposal:
> we declare that a parent task cannot start any child tasks unless it first creates a place for the children to live
^-- This is not satisfied by your method, at all.
> But the problem with this is that you have to know up front the complete list of tasks you're going to run, which isn't always true.
Literally the next sentence after the one you quoted.
> What if you really do need to write a function that spawns a background task, where the background task outlives the function itself? Easy: pass the function a nursery object. There's no rule that only the code directly inside the
Notice how you can avoid the 'setTimeout(() => { .. })` in nested code issue by explicitly passing in a context that has its own scope.
...
There's a lot of stuff in here that is actually quite thoughtful.
You don't have to dig through it, finding every little nitpick you can to call it out on; just think about the idea being proposed.
It's certainly not 'just promises'; if you think it is, then... I don't know what to say. You're wrong. :P
I think you're overestimating nurseries here. They still don't handle the case you described of a deeply nested setTimeout:
import asyncio
import trio
import threading
async def my_tricky_function():
threading.Timer(5, lambda: print('first')).start()
await trio.sleep(.5)
async def a_function():
async with trio.open_nursery() as n:
n.start_soon(my_tricky_function)
print("finished")
trio.run(a_function)
Note that you could imagine that this timer is created in an awaited function or a supposedly synchronous child function.
If the nursery did what you think it did, which is to wait until everything async within the block completes, you'd see this print first, then print finished. It doesn't though, finished is printed first. The reason is that you need to opt in to the trio constructs by using promises/futures/coroutines. Using threaded callback based things give you the same problems with trio as they do in other async/parallel constructs.
If you're a good citizen and opt in to the safety guarantees trio provides, it can keep things clean for you, yes. But you do need to explicitly opt into the scheme by only using code that you know is promise/future/coroutine based. Threads are a doozy.
But the same thing is pretty much true with async/await promises/coroutines in js or python. If you require that you only use promises (or you tightly wrap all of your threads/callbacks in promises and then use that), you can get the safety of nurseries in most of the situations (I noted there were some exceptions before!), but with just the async/await syntax, no need for the extra async context manager.
I agree that trio is cool. I've used it before. But you're ascribing to it and this construct magical powers that they cannot and do not have.
sigh~
In this implementation, or perhaps, in python at all...
Does that make it completely valueless?
I would venture to suggest that maybe there's a big wide world of languages that actually support controlling how threads are spawned, where it might not be.
Maybe its worth considering.
I agree that it's worth considering other ways of handling concurrency. But we should do so by staying within the realm of reality.
Your understanding of nurseries, conceptually, does not match their capabilities. It's not about any language or implementation, it's that what you think they can do isn't possible. It violates the halting problem.
You can't statically infer whether or not a function will have async side effects without opting into some scheme that describes those effects. If you do that, nurseries provide some guarantees. But those same guarantees are provided by just using async/await, which tracks explicitly which functions are async and which are not.
Nurseries do potentially provide some advantages when dealing with tasks that you want to outlive their scope, which async/await doesn't handle well, and when dealing with an unbounded number of async calls (maybe, I think an `async for` construct handles it too).
But otherwise, most of the advantages you seem to think nurseries provide aren't. And not just by this implementation, but by any implementation. They are provably not providable by any implementation that isn't equivalent to marking your async functions as async.
If you can control how threads spawn, like C# thread contexts (for example the MVC context forces async resumes on the original thread rather than an arbitrary one), this is entirely plausible without an 'opt in' model.
This is an opt in model.
In general, it is not possible to know if an arbitrary thread will halt. That's the halting problem. You can restrict yourself and make sure that all threads 'return' on completion (promises, futures), or do various other things. But in an unrestricted system, it is provably impossible to know whether or not an arbitrary thread will halt.
The halting problem is irrelevant to this discussion.
Well.. whatever. You can provably determine whatever irrelevant point you want I guess.
If a naively spawned task or routine can be artificially restricted at a boundary point, that's enough for me.
You could without question implement this by binding child tasks to the threadlocal context on spawn and wait all on a dispose block in c#; certainly if you explicitly went out of your way to swap to a different sync context it would break but so what?
You can always unsafe your way into a hole in any language.
Ok, and you have three options to do this:
1. Block 2. Opt into some system that allows you to defer blocking until later 3. Know when the callback will finish
1 isn't async, 2 is an opt in system, and 3 has a prerequisite of the halting problem.
That's it. You can restrict yourself to not using primitives like `Thread`, and instead only use futures or async/await style things, and that's a valid solution. Hell, your language could not expose a Thread primitive at all. But its solution #2. You cannot just take a look at arbitrary code and pick a point or points where thread execution is restrained.
If you can, it's because the language you're using prevents you from doing certain things. (for example, opting in to "all threads must be promises, and all promises must be awaited)
> The unbridled use of the go to statement has an immediate consequence that it becomes terribly hard to find a meaningful set of coordinates in which to describe the process progress. Usually, people take into account as well the values of some well chosen variables, but this is out of the question because it is relative to the progress that the meaning of these values is to be understood! With the go to statement one can, of course, still describe the progress uniquely by a counter counting the number of actions performed since program start (viz. a kind of normalized clock). The difficulty is that such a coordinate, although unique, is utterly unhelpful. In such a coordinate system it becomes an extremely complicated affair to define all those points of progress where, say, n equals the number of persons in the room minus one!
[0] http://www.u.arizona.edu/~rubinson/copyright_violations/Go_T...
Most code out there is fine using break and continue in loops, and early returns are pretty popular. All of these are unstructured programming (by the 60s definition). I don't think it's all that clear that it has won.
"In the end, modern languages are a bit less strict about this than Dijkstra's original formulation. They'll let you break out of multiple nested structures at once using constructs like break, continue, or return. But fundamentally, they're all designed around Dijkstra's idea; even these constructs that push the boundaries do so only in strictly limited ways. In particular, functions – which are the fundamental tool for wrapping up control flow inside a black box – are considered inviolate. You can't break out of one function and into another, and a return can take you out of the current function, but no further. Whatever control flow shenanigans a function gets up to internally, other functions don't have to care.
This even extends to goto itself. You'll find a few languages that still have something they call goto, like C, C#, Golang, ... but they've added heavy restrictions. At the very least, they won't let you jump out of one function body and into another. Unless you're working in assembly, the classic, unrestricted goto is gone. Dijkstra won."
I tend to agree with the author. Linux kernel code, for example, uses goto for error handling, particularly when implementing system calls. But the author is correct: the gotos can't jump out of the function. Readers can still infer control flow from function call sequences. I quite enjoyed the post, I think it's worth giving the author's claims serious consideration.
1) You should be able to declare scoped blocks that mandate execution of all tasks started in that block ends when the scope ends.
2) This is fundamentally superior to all other forms of concurrency.
I get it; this is basically what async/await gets you, but conceptually you can spawn parallel tasks inside an awaited block, and know, absolutely that all of those tasks are resolved when the await resolves.
(this is distinct from a normal awaited block which will execute tasks inside it sequentially, awaiting each one in turn).
...seems like an interesting (and novel) idea to me, but I flat our reject (2) as ridiculous.
Parallel programming is hard, but the approach from rust, to give you formal verification, instead of arbitrarily throwing away useful tools seems much more realistic to me.