Why? I find when you're importing external data or user input, that's exactly where you want strong types as that's the most likely place unexpected values are going to be generated (e.g. unexpected null values, string instead of an array).
As for data validation, the hard part of it is not checking type but that the value is safe and coherent. And dynamic languages excel at that because they make complex declarative schema descriptions easy while powerful.
See marshmallow for Python: https://marshmallow.readthedocs.io/en/latest/
It will validate everything, including nested recursive data with cross references in fields and a need for sanitation and casting. But you don't need a lot of code to do so.
Static languages are good to enforce safe practices at scale . That's why they are popular in big IT structures. Dynamic languages are fantastic when you have a lot of external data manipulations. Hence their popularity for the Web, banking, biology, geography, etc.
You mean a dynamic check? It's better than nothing but I don't exactly want my e.g. scheduled data import process crashing because it was sent some bad data. I'd much rather have static checking and have all the possibilities accounted for before the code runs.
> As for data validation, the hard part of it is not checking type but that the value is safe and coherent. And dynamic languages excel at that because they make complex declarative schema descriptions easy while powerful.
Why couldn't you write this with static checks? Statically checked functional languages for example are great for writing compilers.
If you have decent type inference there isn't extra typing and any time you think you're saving with dynamic languages gets eaten up having to manually fix things when errors happen at runtime. I'd much rather spend time fixing predictable compile time errors than unpredictable runtime errors.
>>> (1,2) > {1: 2} True
>And dynamic languages excel at that because they make complex declarative schema descriptions easy while powerful.
Marshmallow looks like it works the same way as existing systems in statically types languages. e.g. Serde in Rust:
This validates at compile time.
In [60]: (1,2) > {1: 2}
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-60-f9d83a4c8f73> in <module>()
----> 1 (1,2) > {1: 2}
TypeError: '>' not supported between instances of 'tuple' and 'dict'And the keyword in my schema validation sentence is "easy".
I love rust, but a low level language cannot be compared in productivity to a high level language for this stuff. They are much, much easier. That's the whole purpose. You trade easiness for rigor.
Let's say we're doing a user registration. In most dynamic languages the JSON body will get parsed into a map. Excuse the fat controller and pseudo-language, but it'll end up looking something like:
func create(conn, params) do
if not Validator.is_email?(params["email"]) do
return error(conn, "email is not valid")
end
if not Validator.min_length?(params["password"], 10) do
return error(conn, "password must be 10 or more characters")
end
create_user(params)
true
end
In Go, if you want this nice and typed, you'd add a RegistrationInput struct with the field mapping tags: type RegistrationInput struct {
Email string `json:"email"`
Password string `json:"password"`
}
Map the user's input to the structure, handling the error: var input RegistrationInput
if err := json.Unmarshal(req.body, &input); err != nil {
...
}
And then write the exact same validation checks.What safety does Go's version buy you?
Now, if we are talking about error handling, the way it is done in Go is not particularly representative. Most static languages would go with exceptions (or maybe checked exceptions in Java). Haskell/Rust/Scala would probably go with a Result type, and then it's more a question of the language giving you the tools to propagate the errors and wrap them (Rust with error-chain is excellent at this kind of thing).
Your code starts by making a few checks on the shape of the data you're receiving and failing earlier if some conditions are not met. Great.
Once these checks are cleared, you now know that you can materialize an object with valid values in it, something the rest of your business code will be able to manipulate. So you instantiate that new object, return it and... all the knowledge about the shape of that data is lost.
The only thing your callers will see is an Object and zero information about what operations or fields are available on this object. They'll have to guess.
With a statically typed language, you don't have such a problem. You have asserted all these properties about the data that came in, you know exactly the class of the object you are creating, why forget all this precious information that will keep your code robust?
In my experience, dynamically typed languages are at an evolutionary dead end and offer close to zero advantages over modern statically typed languages (e.g. Kotlin, Swift).
I'm not really seeing how that necessarily falls out of the example given by GP. Could you go into that in more detail?
Is there something preventing create_user(params) from emitting an object with a well defined interface?
If you call that function, you have no idea what the type of that object is. Only the function knows it and that information is lost as soon as the function returns.
And yes, you really have no idea what the type of that object is unless you look at the source of the function. Which is obviously suboptimal, especially if you don't have access to that source.
A statically typed version guarantees you haven't made some fundamental mistakes and enables automatic refactoring.
Say we wanted to change an ID field from being a string to an int later in development for example. You've now got to hunt around all your files making sure the right type is being used. It feels incredibly primitive when you need to do this manually or have to rely on an exhaustive and annoying to maintain test suite for this.
I do like Erlang's approach with hierarchical error domains ("processes" and "supervisor trees"), and I honestly think Go made a mistake here: The semi-recent introduction of Context acknowledges that a control hierarchy is present and an important part of a concurrent, distributed application and that there are aspects of this that need to flow through everything. The Context exists to permit a caller to call functions and launch goroutines that inherit deadlines, the ability to cancel their work, and so on. But this is a rather blunt and intrusive tool that could have been better solved by baking termination into the goroutine scheduling runtime. Of course, one reason goroutines cannot be forcibly terminated is because everything is mutable all over the place, but you could still have termination induce system-level errors that the goroutine would be forced to handle (or maybe you could safely induce a panic)... but that ship seems to have sailed a long time ago.
And yet a hierarchical approach similar to supervisor trees would have been awesome to have in Go. Managing hordes of complex, "crash only" goroutines in Go gets gnarly fast, and you tend to end up with your own ad-hoc type of supervisor trees that uses waitgroups and maps and what not to corral trees of composed workers to ensure they are all able to cancel, retry + back off, and time out appropriately. Even something like retry with backoff isn't part of the standard library, even though it's extremely core to distributed, concurrent programming.
I think I agree with Go's authors that exceptions are bad, but exceptions in an explicitly hierarchical, supervisor-type process model, where processes are "crash only", actually make sense. It means you tend to design your system into discrete parts consisting of dumb workers (that give up and crash on the smallest error) and slightly less dumb supervisors (that react to said crashes). It's not a panacea, but it's better than the loose, somewhat impoverished set of tools that Go gives you.
I really like Elixir and the Erlang VM but supervisor trees are not a perfect solution either. For all the talk about how in Erlang you just let things fail and don't have to do defensive coding, any non-trivial application requires a lot of very thorough design on how to handle error scenarios and keep your application consistent.
If a process fails too many times too often (say your web service it's trying to ping is inaccessible) it can take out your whole supervisor. If that supervisor is holding an ETS table that holds important coordination data (but not data that requires persistence) that ETS table (and all it's data) is now completely gone, and unless that case has been handled you have a lot of processes running but not doing what they should be doing, and your application is inconsistent until you completely shut it down and restart it.
And if you don't plan for a lot of these scenarios then you will be caught completely surprised when it happens in production.
Also you have a lot of control of your supervision tree. You can specify the quantity and frequency of failures that would cause a supervisor to fail. For some use cases, 3 failures in 5 seconds might be reason enough to restart that supervisor. For other use cases, a much higher failure rate might be justified.
These decisions are not trivial and are exacerbated because of the Actor style of flow due to the independent pieces.
So from my experience you have non-BEAM languages where a single error can take down your whole application if you don't plan your failure scenarios properly, where as in BEAM languages if you don't plan your failure scenarios properly you are more likely to end up with an inconsistent system. The former isn't that bad in a lot of circumstances as you should have an external application monitoring and restarting a critical server that's no longer running (even for BEAM systems) but the latter can be very bad because it can be a silent issue.
I have systems working at my job written in .net that have been running for over 6 months without a restart, even when Azure had failures and it could not talk to its command and control server for an hour and kept running perfectly fine with using a well thought out architecture and a well placed central error handler for generic errors without things cascading throughout the whole system.
So resiliency has less to do with language or supervisor trees and more to do with making every step has failure scenarios thought about and planned for.
You are correct that this is how it's often done - 9 out of 10 errors probably ARE "handled" by a central error handler. But this is an incorrect approach if the goal is a reduction of catastrophic errors.
In the following paper, "Simple Testing Can Prevent Most Critical Failures" (OSDI '14). the authors discuss how some catastrophic errors occur in distributed systems that they test (Cassandra, HBase, Redis, HDFS, MapReduce).
http://www.eecg.toronto.edu/~yuan/papers/failure_analysis_os...
Notably section 4.1: Trivial Mistakes in Error Handlers:
"""
Figure 5 further breaks down the mistakes into three categories: (i) the error handler ignores explicit errors; (ii) the error handler over-catches an exception and aborts the system; and (iii) the error handler contains “TODO” or “FIXME” in the comment.
25% of the catastrophic failures were caused by ignoring explicit errors (an error handler that only logs the error is also considered as ignoring the error). For systems written in Java, the exceptions were all explicitly thrown, whereas in Redis they were system call error returns.
"""
I haven't used Go much but I personally much prefer Rust's error handling which appears to be largely similar.
As for infrastructural software, yes you want to handle errors in a more thoughtful manner. If you can't and the process has corrupted it's own stack or heap (in the case of C or C++) then sure, abort. But I'm saying this line is often chucked over to "log the exception and do nothing" too quickly because it's often the easier thing to do. 25% of catastrophic failures in this one study appear to occur because of this.
tl;dr: The devil is in the "If you can't".
* Unfortunately error handling sucks (no stacktrace by default? Let's go back to the days of peppering print statements around just for fun).
* DB support is terrible (I've spent hours doing what I could have done in half an hour with python).
* It's so verbose, even simple things like mapping over an array to create another array requires a for-loop dance. Same for just checking if a value is in an array. Same, obviously for no exceptions which you either love or hate. I hate.
I can't see myself doing any other projects with it.
goroutine 1 [running]:
main.b()
/home/rjp/git/godiggy/test.go:8 +0x64
main.a()
/home/rjp/git/godiggy/test.go:13 +0x9b
main.main()
/home/rjp/git/godiggy/test.go:18 +0x9b
exit status 2Please, nobody write full fledged WebApps or all of your middleware in lua. Trust me that's a horrible idea. I have seen about everything you can do in lua/nginx and it does not turn out pretty.
Is this a happy-hour for Reddit users on HN?
This hasn't been my experience. I work in Python (and JS) by day, and I can't wait until we upgrade to 3.6 so I can reasonably employ type annotations. Without them, people tend to write hacky code, such as functions which do completely different things based on the data type passed in (rather than fixing the abstraction) or build up huge, nested data structures with no formal definition (various parts of the code base just tack on whatever they like, and the documentation inevitably falls out of date). Even my prototyping is in Go, because I would rather have a compiler catch my errors than to spend tenfold the time trying to debug. Also, when I'm working in Python, I sorely miss the tooling that I have in Go and other static languages.
I don't think Go is the end-all-be-all of programming languages, but I think any dynamic language would be more productive if it was at least optionally typed.
You can still utilize all of your cores in Nodejs. So which problem are you encountering?
My opinion is the opposite. Almost every go code base I've seen is extremely consistent compared to other languages. Going from one code base to another is almost always seamless because learning Go involves learning the Go tools which forces you to follow Go coding standards. I'd choose to work on a Go code base over Java or C++ any day of the week.
I don't have much experience with elixir, but that's because I like to stay employed.
>the entire error thing is absurd. Anders Hejlsberg got this right many years ago: 9 out of 10 errors are "handled" by a central error handler
This is almost always unacceptable in production, especially when it comes to mission critical hard/software. You need to be able to handle errors in production that you may not even be able to catch in testing and that means you need to catch all the errors and think critically about what should be done when that error is encountered. "Central error checkers" are almost never robust enough for mission critical systems.
Go basically forces people to write high-quality, resilient code bases, and I like it for that.
This is not true for almost all software outside of embedded domains. In fact responding to an error near where the error occurred is almost always a mistake. The rare exceptions to this are when "error" is unavoidable due to concurrency (e.g. I/O), and even then many errors shouldn't be handled (misconfiguration, incomplete system initialization, etc.) or should be wrapped and transformed for human consumption (e.g. user tried to access a resource that couldn't be found - we still bubble up).
> you need to catch all the errors and think critically about what should be done when that error is encountered.
If you support this position, you'd support checked exceptions like Java - something that's widely considered a mistake.
There's lots of reasons why it's a mistake. A more important reason has to do with dynamic construction of control flow. If you use any monad design patterns in the construction of your application (easy to do because the monad design pattern is so powerful), you quickly create boundaries that errors need to cross unmolested because they're generic and cannot possibly encode policy. The same criticisms follow through into dependency injection and other forms of modular decomposition that use dynamic recomposition. Java style has you doing utterly pointless things, like wrapping all the implementation errors in a generic module-level error - obscuring the very thing that might theoretically let user code react to the error state!
More philosophically, abstractions fail in implementation-dependent ways. When you encode policy about failure outside the abstraction boundary, you're encoding implementation-dependent behaviour; but you can't put the policy inside the abstraction boundary either, because then the abstraction isn't reusable - different applications have different policies. Error handling is inherently opposed to abstraction. That's why microscopic error-handling close to the cause of the error only flies in small closed systems, like kernels, databases, embedded.
The primary problem of checked exceptions in java is not that they exist, it's that they are very badly integrated with the rest of the language e.g. it's impossible to be generic or transparent over checked exceptions, their hierarchy is messy and it's often unclear why specific exceptions are checked and others unchecked. Haskell's error handling (reified through Either error values) has similar effective semantics but nowhere near the dislike because it's well integrated in the language and you can be generic over the error, the value, and the result itself.
Generic composition that lets error types flow through abstractions also lets implementation details flow through, so it is an architectural choice, make no mistake. You don't dodge the tradeoff between abstraction and error handling.
But, that doesn't really have anything to do with it being a chore. We can just agree to disagree on that.
However, I think you misunderstood my point about errors (or maybe you're just working with very different types of systems). Empirically, looking at Java codebases, the C# team found that most errors weren't "handled" beyond an outer exception handler. Anecdotally, this has been my experience as well. I admit that this is a very 'web' point of view, where requests are isolated and most exceptional situations are best handled with logging + showing the user a friendly error message.
At this point, they should adopt the Elixir approach and consistently provide a ! alternative across the stdlib (Must* is inconsistently available and verbose). They thought Go would be used by C++ programmers, instead it's being used by Ruby programmers (1). The language paradigm doesn't fit how people are using it as well as it could.
I'm not sure what other languages you've been exposed to, but your last line is hard for me to believe. If you look at other languages, you'll find a lot of techniques aimed at quality and resilience that are nowhere in Go. Again, that doesn't make Go bad, but "resilient" ? Go has mutable, shared memory, nils, no thread isolation, no generics (so runtime casting), no pattern matching and so on. Even the error system which you like is reckless (??) compared to what you'll find in languages like Rust and Haskell (Result and Maybe).
(1) https://commandcenter.blogspot.com/2012/06/less-is-exponenti...
While the previous commenter gave an objective and rational overview why he think Go gives him a hard time you can't resist to get polemic.
Maintaining Cobol code is a chore, though, partly for the same reasons as Go code.
Same message and nobody loses his face.
Yeah, because Go is the epitome of employability...
>"Central error checkers" are almost never robust enough for mission critical systems.
Go is not used in mission critical (embedded, medical, aviation, communications, etc) systems anyway.
Besides the notion is wrong: mission critical systems DO centralize error checking. The code local to the error wouldn't have enough context (and shouldn't have the reach) to call the right procedures for cleaning up and moving on.
> This is almost always unacceptable in production, especially when it comes to mission critical hard/software. You need to be able to handle errors in production that you may not even be able to catch in testing and that means you need to catch all the errors and think critically about what should be done when that error is encountered.
Absolutely. My approach is usually that errors are always wrapped not just returned. The wrapping itself ends up looking almost like code documentation: "could not do X: <previous error>", "error while doing X: <previous error>". Every error string is then unique in the program, easy to grep.
"Could not perform the operation", maybe together with a 500 lines stacktrace in your logs (which are maybe line-by-line bacause that's the default)... you are up to a nice middle of the night debugging session :)
In general, I found it a chore to maintain Go-based systems.
There's a lot to like about Go. But it doesn't seem pragmatic for the types of applications I see people using it for.
For example, the entire error thing is absurd. Anders Hejlsberg got this right many years ago: 9 out of 10 errors are "handled" by a central error handler (log + "sorry, try again"). You can do this with error values, but at best it's verbose. Maybe error values makes sense for system programming, I could certainly believe it makes more sense.
But, really, the main problem is the type system. I'm a huge fan of dynamic languages. I think they're so much more productive. This is especially true when your system is interacting with external data - like user input and a database. I did C# and Java for years. These interactions are, at best, painful with static languages. But they're downright infuriating in Go (poor reflection, no generics). I wrote a somewhat popular library to help deal with map[string]interface{} nonsense (2), but it's not enough.
When it comes to web/tcp stuff, I think OpenResty for any middleware-ish system and Elixir for everything else are far more productive and enjoyable ecosystems. We've been doing Elixir for about a year now, and I've never seen such a consensus on productivity and quality (everything else always had some issue: Ruby is slow, Node is single threaded, Java is verbose).
There's certainly more space than just that. Go might still be a good choice for CLI tools, and both OpenResty and Elixir have a distinct memory/threading model that precludes some use cases - but those are in the minority.
(1) - Nothing against Erlang model, but I do think it's more restrictive.
(2) - https://github.com/karlseguin/typed