It’s a logic bug, but you can’t “not panic”. You can trap and recover it though.
Bit orthogonal to OP but relevant to your reasoning.
If you have any care for quality at all, there's nothing "simple" about your invariants being violated.
On a memory safe language, the blast radius is potentially much smaller (in principle all objects transitively reachable from the failure point, but this is very pessimistic), so it is realistic to be able to isolate the failure in process.
Crashing the program is strictly worse imo, since there are other concurrent requests which will now fail for no reason.
I agree unhandled panicking makes sense sometimes. It’s context dependent.
Program state is a good argument, but I think it really depends.
_some_ part of a program's state has become invalid. Probably a variable on the stack. In web servers, almost always a part that becomes very irrelevant as soon as the current request ends. In databases, the same is true at transaction boundaries. In shells it's the REPL. Exit(1) in any of those deep down the stack? No thank you, wouldn't touch that software with a 10 foot pole.
No, I write bugs all the fucking time! That's why I want my code to panic, so I know I wrote a bug. If that shit gets stashed away in a 500 response in a Prometheus metric maybe I'll chance to see it three days later. But if the container gets blown away, I'll notice.
I heavily used errgroup before creating conc, so the design is likely strongly influenced by that of errgroup even if not consciously. Conc was partially built to address the shortcomings of errgroup (from my perspective). Probably worth adding a "prior art" section to the README, but many of the ideas in conc are not unique.
> In go land, this seems desirable.
I mostly agree, which is why `Wait()` propagates the panic rather than returning it or logging it. This keeps panics scoped to the spawning goroutine and enables getting stacktraces from both the spawning goroutine and the spawned goroutine, which is quite useful for debugging.
That said, crashing the whole webserver because of one misbehaving request is not necessarily a good tradeoff. Conc moves panics into the spawning goroutine, which makes it possible to do things like catch panics at the top of a request and return a useful error to the caller, even if that error is just "nil pointer dereference" with a stacktrace. It's up to the user to decide what to do with propagated panics.
This is exactly what's undesirable. (IMO and from my reading the GP agrees.)
The problem is that panics aren't "goroutine scoped" in terms of their potential impact. So it really shouldn't be up to the user to decide how to handle a panic. Application code shouldn't handle panics at all! They're not just a different way to yield an error, they're critical bugs which shouldn't occur at all.
I'm with ya there. However, there are also many classes of logic errors that are not goroutine-scoped. And there are many panics that do not have impact outside of the goroutine's scope. In my experience, this is true of most panics.
In practice, panics happen. They are (almost) always indicative of a bug, and almost always mean there is something that needs fixed. However, if a subsystem of my application is broken and panicking, there's a pretty good chance that reporting the panic without crashing the process will provide a better end user experience than just blowing up.
Yes, that means I'm accepting the risk that my application is left in an inconsistent state, but coupled with good observability/reporting, that's a tradeoff I'm willing to make.
(bonus: this is especially true when propagating panics allow me to capture more debugging information to fix the panics faster)
I guess this is the crux of the issue. I don't think this is true, or needs to be true. It certainly hasn't been my experience. I think assuming panics are normal will take you down some paths that make it basically impossible to write reliable software. But, to each their own.
> I'm accepting the risk that my application is left in an inconsistent state,
Inconsistent state makes it impossible to reason about your program's execution or outcomes. An account value that previously had balance = 0 may now have balance = 1000. Is this acceptable risk?
Runtime-safety "panics" in Go, like concurrently modifying and iterating a map that can lead to other memory being corrupted, tend to abort the whole process immediately and not be suppress-able panics.
https://go.dev/doc/effective_go#panic
> The usual way to report an error to a caller is to return an error as an extra return value. . . . But what if the error is unrecoverable? Sometimes the program simply cannot continue. For this purpose, there is a built-in function panic that in effect creates a run-time error that will stop the program
Panics express unrecoverable failures. This is plainly stated in the language documentation. There are exceptions to this rule, but they are exceptional.
All safety guarantees possible if there were no panics are possible with.
Na, citation needed. Assuming "panics are normal" is just extrapolating from "errors are normal". It makes reliable software more reliable.
Which would definitely impact the quality of the software overall in a bad way.
* Make sure you support properly unrolling the stack
* Keep a clean failure boundary, probably somewhere on top of your loop
* Fastidiously check your preconditions
* Fail brutally if they're not met
* Improve from there
This is trivially falsifiable by panicking yourself and immediately recovering. Neither failure domain nor failure boundary need to align with the entire process.
func (x *Thing) Method() {
x.somethingThatPanics()
x.somethingThatAssumesTheAboveDidntPanic()
}
You're adding preconditions to your claim.
> the entire program state is invalidated.
No, the state represented by a connected graph of variables accessible by the concurrent actors is tainted. This is hardly "the entire program state". Often it's just a few cache entries.
Also, see my first, most important, bullet point:
"* Make sure you support properly unrolling the stack."
Which means a request to Thing errored out, but it never enters an invalid state. If you fail at that, all bets are off. But then you're dealing with a mediocre codebase anyway.
And finally, let me rewrite your example to something, that I see much more often in real life code, which problematic _even without concurrent actors_ because somethingThatAssumesTheAboveDidntReturnAnError might do horrible things all by themself:
func (x *Thing) Method() {
x.somethingThatReturnsAnError()
x.somethingThatAssumesTheAboveDidntReturnAnError()
}
Not inherently, but it's your job as a developer to make sure this is the case, that's what:
"* Make sure you support properly unrolling the stack."
means.
In case you're dealing with unknown code it's your job to find out what the connected graph of potentially tainted objects is and discard them. That's what "keep a clean failure boundary" means.
If you can't, because you don't want to (short lived process, prototypes) or are unable to (hairy ball of code), tearing down the process is indeed the only option and a sane fallback choice made by the language designers. But it's not necessarily a hallmark of robust software.
I hope I had a final shot to clear up what I meant, thanks for the discussion anyway.
Your entire web app process crashes due to a panic every time a request triggers an extremely rare edge case. A hacker discovers this and uses it to conduct a DoS attack. Is this acceptable risk?
// RED FLAG
x := y.(type)
// good
x, ok := y.(type)
if !ok { return an error }(joke)
Of course, a language with less runtime errors is a far cry from being a panacea. Avoiding runtime errors is not the same as avoiding all categories of bugs. And while I personally prefer stronger type systems - they definitely come with increasing levels of cognitive cost.
But I still feel that the type-safety vs. runtime trade-off is more often ignored, underestimated or undersold than it is being hyped. Yes, certain languages (cough Rust cough) are being hyped, but not the conscious choice of balancing programmer learning curve with runtime type-safety.
And while on the topic of Rust, it's probably not the best choice for a language that sees less runtime panics. Especially since unwrapping an error is always the easiest way to handle an error, and thus quite common. But lazy error unwrapping aside, Rust does avoid null dereference exceptions, type casting exceptions and most types of race conditions that can be quite prevalent with go[1].
Sure, if we could write code without bugs, we wouldn't need to suppress panics. But since we do tend to write code bugs and some of them are bugs that can be detected by the runtime - we get panics.
If you hate panics, you can do better than Go and go for a language with a stronger type system, where you won't get nil pointer panics or interface conversion panics, but even an almost onerously-tyepesafe language like Haskell still panics on some bogus operations such as division by zero or trying to read the head of an empty list. Perhaps Idris really have no runtime errors but they are quite niche.
I'd rather have the control to log the panic on a service rather than it forcibly dying and taking down any other connections with it. Kube will just spin up a new one anyway, which just introduces a downtime gap that doesn't need to exist.
I do not think you are correct. Go has a class of unrecoverable panics for this specific reason. Go also runs deferred functions after a recoverable panic, so the notion that it's unsafe to handle it, or continue executiona after doesn't hold at all - it is literally a first-class feature of the language.
I have not seen an instance of a recoverable panic that is raised _after_ such a fatal operation. If you have an example of such, I would love to see it.
As far as the user/developers are concerned, it doesn't matter too much, since you have no option to recover them, but it would be nice if it was explained if defers are still ran. I'm assuming they are not.
1. https://stackoverflow.com/questions/57486620/are-all-runtime...
Of course you cannot distinguish between unrecoverable and recoverable panics, because by definition an unrecoverable panic is not recoverable. There is no caller to distinguish between it - it is killed.
The point I'm trying to make is that panics are not errors by another name, and they are not safe to recover from in general.
Index slice out of bounds? panic. Close a channel twice? Panic. Incorrect type assertion? Panic. Dereference nil pointer? Panic.
I would argue that all of these examples which are the most common in my experience are “goroutine scoped” because the goroutine was aborted before they potentially modified the application state in an unknown way.
It’s like not in C, or C++ where out of bounds access has now put the entire application into an unknown state.
These are all really bad things which should never survive to production code. It is not difficult to detect and prevent them.
> I would argue that all of these examples which are the most common in my experience are “goroutine scoped” because the goroutine was aborted before they potentially modified the application state in an unknown way.
What makes you think that terminating the goroutine that triggered these panics prevents them from impacting the process state?
> It’s like not in C, or C++ where out of bounds access has now put the entire application into an unknown state.
What makes you think this is the case? Panics have unknowable impact, and many panics (e.g. data races) absolutely do put the program into an unknown state.
This is equivalent to saying "out of bounds memory writes are not difficult to detect and prevent in C code". Like actually equivalent (possibly worse), not just "well if you squint they look similar".
Of course it's not hard most of the time. Being perfect is beyond hard though. And if you're not perfect, you might open the door to anything in C, or cluster-destroying rolling crashes in Go.
Sometimes shutting down every piece of your software if that happens is the correct choice, and sometimes it's so far beyond reasonable that it's ludicrous to argue in favor of "every panic is an abort".
Very much this. And even for the same project: in some cases, I'm a fan of employing a quite strict error handling policy in dev environments (crash and burn) and using a more lenient approach in prod (elevated log level). In my experience, this can result in a robust product. Most importantly, this means the decision is not even made by the application programmer, sometimes it's a config thing.
How do you "detect and prevent" all nil pointer dereferences?
this kind of thing is why deepcompare exists to begin with
They are not inter-changeable, nor can one replace the other.
> If the key type is an interface type, these comparison operators must be defined for the dynamic key values; failure will cause a run-time panic.
I can find a mention of "cmp.Equal" having that behavior, but that's just a third-party package panic.
Many (perhaps even most?) data races would not result in panic but just in garbled, missing, duplicate, out-of-order or otherwise incorrect data.
Data-race induced panics are generally the side-effect of a data race, not a direct protection against. They can often be inconsistent: e.g. a data race in a slice that contains a binary data format could garble a variable-length string prefix and produce an index-out-of-bounds panic. Or it could prematurely consume a shared pointer and overwrite it with nil, only to have the nil pointer dereferenced by another goroutine. These kind of panics are unpredictable.
If your application has shared global state (in-memory or even a database), it may become inconsistent due to data races. But whether data-race induced indicate irreversibly corrupted global state that requires (and can be fixed with) application restart - that is case-by-case thing.
Let's say your application has some shared state that got corrupted and the corruption triggered a panic down the line.
If your shared state is persisted in a database or some other distributed mechanism and that state got corrupted: restarting the application won't help you.
If your shared state is scoped at the HTTP request level (or whichever boundary you choose for suppressing your panics): you don't need to restart the application. The request is already terminated, along with its shared state.
Which leaves us with in-memory global state. This kind of state is generally minimized in the type of microservice and network infrastructure applications that Go is often used for.
A very small percentage of your panics will indicate corruption of such state. Will you be willing to risk service downtime in order to protect against the small possibility that the service has run into a state where its shared in-memory data became corrupted?
in memory or some other distributed mechanism and that state got corrupted: restarting the application won't help you.
Production systems should have relatively robust testing whose coverage can be increased over time. When something panics, the cause of the panic should be fixed so that the panic never happens again. Over time panics shouldn't be happening.
Then again the systems that I have relied on I have written on my own without other hands in the pot so maybe I just don't have to deal with the reality of other programmers phoning things in.
This would be fine for a low-level language like C where you need to allow SEGFAULTs, but designing it into high-level language this makes no sense.
And there are some operations where you cannot check in advance whether a panic will happen: comparing interfaces (underlying values might not be fully comparable), indexing a map (could blow up during any concurrent write), and closing or sending to a closed channel.
Other paradigms, like the Erlang paradigm, can have better behaviors even if a top-level evaluation fails. But in an imperative language, there really isn't anything else you should do. It is arguably one of the Clues (in the "cluestick" sense) that the imperative paradigm is perhaps not the one that should be the base of our technology. But that's a well-debated matter.
(Haskell’s lazy evaluation just makes it a bit harder to catch, since you need to force evaluation of the thunk within the catch statement, and it’s far too easy to end up passing your thunk to somebody who won’t catch the exception.)
As a matter of Go style, of course, you should almost always defer unlock() after you lock(), but some people sometimes get clever and think that they can just lock() and unlock() manually without using defer. This is not hypothetical, and it causes other problems besides leaving dangling locks after a panic(). Somebody sticks a “return” between lock() and unlock(), without noticing, for example.
So my impression of catching panic() is that it is about as safe as not catching panic(). What I mean by that is that if recover() is not safe in your code base, there is a good chance that there are other, related bugs in your code base, and being a bit more strict about using defer and not trying to be clever will go a long way.
Unfortunately this was not the case before 1.14, so there's a lot of "middle-aged" code floating around setting a bad example.
lock.Take()
thingThatMayCrash()
lock.Release()
However, "access an array element out of range" isn't the question. The question is, what can the language do in the face of any exception? Haskell, perhaps ironically, doesn't avail itself of the opportunity to do something useful about it. You can implement a wide variety of mechanisms that are safe to have exceptions in even in the face of concurrency, but at the base language, it is essentially as exception unsafe as Go, exactly as you observe. (And you can create "exception-safe wrappers" in Go as well, as easily as having an intermediate function that does something with panics, then invokes some code. Nominally this is unsafe because the code being invoked really ought to have a guarantee that the user has written it to be safe in this usage; in practice it works fairly well at significant scales due to a combination of other things beyond the scope of this already-large reply.) This causes significant practical stress within the community and probably would be in the Top 5 wishlist for a lot of people as to something a sequel language would fix. (It is very annoying that the supposedly "pure" code "head []", supposedly of type "[a] -> a", throws an IO exception.) Just as in Haskell you can write safe code, you can write safe code in Go here as well.
lock.Take()
defer lock.Release()
thingThatMayCrash()
Thus it is not a coincidence that when I named a language that has a different paradigm that allows it to systematically recover from this sort of fault, I named Erlang, not Haskell. Erlang has no locks. (At the Erlang level. NIFs may do their own thing but they are basically a form of "unsafe" and for language analysis should be treated as such.) Since it has no locks it can't blow up the way the Go code can and leave dangling locks. Processes get access to resources that need to be cleaned up through the IMHO somewhat confusingly named "ports" system, which can be conceived of as wrapping up sockets and files and other such resources behind other Erlang processes, and then there's a linking system that says "if this process crashes, send this message to that other process or crash it also". So an Erlang process can safely crash, release all associated resources, and the runtime may continue on with the assurance that there's no dangling locks or other concurrency things half done.
In fact in my opinion Erlang isn't even a particularly "functional" language. This is idiosyncratic and I don't deny it. A better way of understanding Erlang is that it was built around this functionality existing, and the ports system and immutable terms are the tools that were used to accomplish this goal, with the fact that the result looked sort of "functional" being an accident. So I wouldn't necessarily highlight "functional" languages in general as being natively good at this sort of handling. I think it's pretty clear it could be added to a Haskell++ without much effort but it is not something that merely "being functional" automatically gets you out of the box.
(Historically, Erlang actually descends from Prolog. While all the logic functionality is stripped out, the heritage is very visible. You can't really call it a "logic language" since it can do no logic, but in my opinion it isn't in the functional stream of languages either. Your mileage will vary; like I said I don't deny this is an idiosyncratic take, but for what it's worth, it's one from someone who used Erlang professionally for many years and knows Haskell fairly well too, so it's not an entirely uninformed one.)
I suppose the core issue is transactions. Erlang could still fail with
them ! {sub, n}
crash()
me ! {add, n}
I suppose instead transactions could be a core language semantic. But hard to do, and erlang actors gives a lot, for very little overhead
In general, whatever it was the crashing process was doing can still have failed. This obviously doesn't magically fix database transactions, or make it so files are never half written because it crashed in the meantime, or any other "external" action that you can still have failed to protect properly from crashes. It only means that the runtime can't get stuck due to locks failing to be released.
But hey, that's something!
It's difficult to come up with examples of when this may be the case. Often it's a "you're holding it wrong" kind of thing. For example, a common idiom is wrapping something like
func Parse(s string) (Foo, error)
func Must(f Foo, err error) Foo {
if err != nil {
panic(err)
}
return f
}
var someFoo = Must(Parse("hey"))
In case you actually are interested in Go's take on stack traces: You are intended to annotate your errors so you can build your own stack traces with whatever information you want. This leads to better error messages because additional context (values of things) can be logged with your custom stack trace.
http GET /abc: start DB: run migration COOL_MIGRATION: invalid foo, given 'bar' important argument: file not exist
The point is that these aren't just Xs on a map telling you where something went wrong, they're supposed to be meaningful descriptions of why something went wrong.
Yes, that's common. Perhaps a gopls search engine for error messages would be useful?
Many errors don't need metadata. For example, io.EOF signaling the end of a stream is a normal condition. But there are those "true" errors which are unexpected conditions indicative of a bug or a scenario that should be handled differently. To make those easier to find, you can annotate the error values to make it clear what the origin was. This is why there's now a well-established convention to wrap with contextual descriptions at every point in the chain.
In about 8 years of working full-time with Go, the number of situations where I've struggled to hunt down the source of an error value is pretty close to zero. Not zero, but close.
Not-so-contrived example: a config options contains a value that leads to a division by zero in some library I'm using. I don't want that to crash my whole program, I want to tell the user "hey this part didn't work" but be able to continue with the other parts.
Without writing a ton of boilerplate code.
> A frequent problem with goroutines in long-running applications is handling panics. A goroutine spawned without a panic handler will crash the whole process on panic. This is usually undesirable.
In go land, this seems desirable. Recoverable errors should be propagated as return values, not as panics.