If Node.js used language X,Y,Z instead of Javascript, it would certainly not have been the success it was. "Javascript on the server" was the marketing/propaganda . Nodejs came at the right moment, Js was exploding, websockets were exploding, coffeescript and co were exploding in popularity.
What Go does better than Nodejs is a better standard library, true concurrency, static typing. With Go it doesn't matter if an operation is blocking or non blocking, that fact can totally be abstracted from the client code. Writing callbacks is tedious, promises are tedious, co routines with yield need plumbing and async isn't in the spec.
> you're probably not qualified to manage this project at scale - Because scaling across multiple CPU cores is kids' play compared to scaling across multiple machines
With Node.js any heavy computation needs to be launched in a separate process, which is not the case with Go. sure you can have one/2/4/8 process dedicated to heavy computation, but it will still be the bottle neck. In the end you'll have to use a different language in order to solve the single threaded problem.
Finally there is no mystery as to why some businesses are moving from Node.js to Go: They didn't need anything Node.js provides at first place. They didn't need to use javascript, they didn't need to write "universal" applications with the same logic running in the client and in the server, they didn't need single threaded async programming.
Go is not a silver-bullet, but I was a bit sick of having to write a callback every time I wanted to make an http request, just to prototype something.
> With Go it doesn't matter if an operation is blocking or non blocking, that fact can totally be abstracted from the client code.
No, it can't, and pretending that it can is misleading in a way that allows large teams of developers to cause themselves real problems.
The only sense in which this is true is that you can write a Go function with a simple signature like this:
func DoSomeStuff (error) {}
But, make no bones about it: this function blocks until it completes. This means that while this function may do concurrent work it is absolutely a blocking function. This is fine: sometimes blocking functions are good. But you cannot write a non-blocking Go function in the same way.
To make a function non-blocking you can return a channel out of it for the return value to appear on, like this:
func DoSomeStuff (chan error) {}
Except...that return value just there? That's a Future. It's a terrible, half-implemented version of a Future, but that's exactly what it is. It's a promise to return some kind of result at some point when the underlying process has returned.
And if you don't want to block your current goroutine, you cannot block on that channel receive either. That means that you need a callback. There are two patterns for doing that: you could have some kind of central loop that selects over all channels like this and calls the callback functions (boy that looks a lot like Node's event loop, doesn't it!), or you can manually spawn your own callback functions in their own goroutines. Either way, you have callbacks and futures here: you're just building them yourself and calling them something different.
There are lots of good reasons to switch to Go: it's a language that makes lots of developers remarkably productive, it has an ingrained philosophy of building concurrent programs, it's pretty damn fast, and it runs on all kinds of awesome platforms. But claiming that Go has learned something magic and new about how to write concurrent software in such a way that you don't have to care whether your code is async or not is just not true: you always have to care.
The caller can make function "non-blocking" by wrapping the call in a goroutine themselves. (There's some subtle differences, but they are mostly irrelevant here). For this reason, I'd say there is (almost) no reason to introduce asynchrony in your API in the way you suggest. The rest of your post built on this example seems shaky to me, since it seems built on an example API that doesn't need to exist.
I'd say that "you don't have to care whether your code is async or not" is a overstating the case. I would append the qualifier "unless you're introducing concurrency". Considering that almost no low-level APIs are asynchronous, this usually happens rarely (or happens in low-level code like the HTTP server). Examples that have come up for me: making N parallel RPCs, writing a TCP server. In those situations, you care about async vs not.
In event-loop based systems, it seems like async is in my face all the time, even when doing things that are entirely sequential.
Sure, but if they want the return value then either they need to construct the Future-y wrapper I just described or they need to assemble it together in a collection of other function calls wrapped inside a function that itself is either Future-y or uses a long-lived channel to communicate results.
It is not novel to build up a non-blocking system from purely blocking method invocations. We've been doing that for years: it's called threading. Doing things this way has many advantages when written with appropriate diligence, and I'm not pretending otherwise. However, if you actually care about communicating between these arbitrary threads of execution than you either need Futures or queues (both of which are essentially just channels in Go), and at this point you've got the exact same problems as you get in NodeJS or any other asynchronous programming environment.
> The rest of your post built on this example seems shaky to me, since it seems built on an example API that doesn't need to exist.
I don't think that's fair: as I mentioned above, the fact that you as library author would not write the Future-y extension doesn't mean that the Future-y extension isn't built: you just force your caller to build it. That's fine, it's a perfectly good architectural decision (probably you should't be making those decisions for your user), but it doesn't remove the problem.
> I'd say that "you don't have to care whether your code is async or not" is a overstating the case. I would append the qualifier "unless you're introducing concurrency".
Sure. The thing that matters here is that Node is always introducing concurrency, because Node is concurrent. This is why all Node programs have to care about concurrency: they are all concurrent because their system is concurrent.
This is desperately inconvenient for many one-off programs, which is why I personally don't use Node for anything like that: I'd much rather use Python or Rust or Go. But that was never my argument. My argument was about OP's assertion that "with Go it doesn't matter if an operation is blocking or non blocking, that fact can totally be abstracted from the client code. Writing callbacks is tedious, promises are tedious, co routines with yield need plumbing and async isn't in the spec."
The first sentence is dangerously misleading (while technically true, any system that does that is usable only in that one context), and the second one misses the point, which is that those things get effectively built anyway in any moderate-scale concurrent system in Go.
But my biggest point is this: Go isn't magic in regard to concurrency, and there is a weird amount of magical thinking around Go. Go is a very good language with a lot to like, and I like it quite a lot. But when boiled down to it, Go's concurrency model is threads with a couple of really useful primitives. And that's great, and it works really well. But it's not new or novel.
The sentence "with Go it doesn't matter if an operation is blocking or non blocking, that fact can totally be abstracted from the client code" is equally true if you replace "Go" with "C", or "Python", or "Java", or any language with a threaded concurrency model. There's no magic here. It's the same building blocks everyone else is using.
As I said above, green threading has advantages over OS threading, but they behave exactly the same in terms of design patterns and potential bugs.
This is what I was getting at when I said "not that different": compared to the difference between event-loop concurrency and threaded concurrency, M:N green threading is basically just a subcategory of threading.
In normal Go, nothing is blocking in the Node sense. (Oh, if you put your mind to it you can manage it, but I've never once encountered this as an practical problem, either in Go or the equivalents you can do in Erlang if you put your mind to it.)
This has profound changes on how you write code.
It's true, Go is not magic. It's just another threaded language in most ways, with the "real" magic in the community best practices around sharing by communicating instead of communicating by sharing. In theory, you could write an equivalent set of C libraries and get most of the same things, but you'd have a lot of library to write. (This is why things like porting C goroutines to C have a hard time getting traction. It can be done, but it's actually the easy part. Also, you'd still be in C, which is its own discussion. But you can get the concurrency.)
The real issue here isn't that Go is necessarily exceptionally strong at concurrency, the real issue is that Node is exceptionally weak. It introduces this new concept of "blocking" that only exists in the first place because it is weak, and then makes you worry about it continuously, to the point that many people seem to internalize the concept as what concurrency is, when it isn't. It's really just something Node laid on you. So when you step out of Node, and you see a community that isn't visibly as worried about "blocking" as the Node community, someone trained by Node thinks they are seeing a community that "isn't good at concurrency". My gosh! Look how cavalier they are about "blocking"! Look how they tell people not to worry about it, and how casual they are about having users wrapping library code in goroutines and explicitly telling library writers not to do the concurrency themselves. But what you're seeing is what happens when you simply no longer have the problems Node and "event-based code" brings to the table. Go is not magic in the general case, but, honestly, when someone coming from the Node world picks up Go, I can see why they might go through a period where they sort of think it is. There are really differences in code style, and how easy it is to write correct code.
You have to make sure you're not letting the limitations of one connotation of "blocking" spill over into the other, or you will have problems. (True in both directions.)
To speak to someone else's point, "futures" in Go don't "suck", they basically don't exist. If you're writing in a recognizably "futures" fashion, you are not writing idiomatic or even particularly good Go. You don't need futures, because (what are today called) futures are basically an embedding of a concurrency-aware language into a non-concurrency-aware language, and you don't need them when the language you're working in is already concurrency-aware. That's why you don't see futures in Haskell or Erlang either. (I have to qualify with "what are today called" because the term has drifted; for instance, Haskell does have explicit support for an older academic definition of the term with MVars, but modern software engineers are not using the term that way.)
Lots & lots of idiomatic go code exists in that form (anytime you wrap a select that times out in a function you have a future).
Channels, what we are really discussing here, have 2 problems: the first is in abstraction, they don't provide basic primitives that other similar structures provide, like timeouts & cancellation. The second is in implementation. As futures you have to worry about all the edge cases around nil & closed channels. As queues they are highly contended.
All I'm trying to do is to make sure that people who make bold claims about abstracting away blocking code aren't misleading others: when calling other code you should always be aware of how it interacts with the flow control of your program.
Are you sure you're not just explaining how to write a Node program in Go? Write Node programs in Node.
To have an abstraction where you really don't care about blocking or not you need promises/futures. Go's futures are bad. Real bad.
If you don't want the function to be non-blocking then you are fine with either method signature.
function isValidFoo(val) {
...return boolean
}
To avoid a ripple effect, all functions have to be designed to be asynchronous from the beginning, but this increases program complexity significantly. It means avoiding built-in language features such as direct return values, and often means substituting async library functions for built-in looping constructs.
Other languages including Go aren't like that. You don't need to wrap a simple return value in a promise or callback just because retrieving it might someday involve I/O. Execution of the thread/goroutine simply resumes whenever the value is ready, and the function returns normally. In those languages, Node's distinction between sync and async is artificial and unnecessary.
You can literally say that about every labguafe that has concurrency abstractions.
The whole point of the argument is that golangs abstractions just push the callbacks (or blocking sync) to the application layer.
To me, idiomatic Go suggests that libraries, data structures, business logic, &c is encapsulated in blocking functions, and that concurrency is expressed in glue code.
The argument to me is that Go advertises itself as being easy to write that concurrent glue code & then fails to provide basic abstractions around it.
Everything you mention about things being expressed in glue code would be easier if Golang provided a modern promises library & holds just as reasonably for other modern languages that do.
The "node requires callback hell" is a red herring because so does Golang (that or blocking code), just it does it at the app layer instead of the framework layer.
What‽ It provides channels, which are a really nice abstraction for concurrency.
> Everything you mention about things being expressed in glue code would be easier if Golang provided a modern promises library & holds just as reasonably for other modern languages that do.
Have you actually written any Go? Using channels in Go is much nicer than using callbacks. Callbacks basically force one to write one's code in continuation-passing style, while callbacks let one work about sync points only where you need.
Channels are a concurrent safe queue, nothing more. They are not even particularly well implemented concurrent queues as they are highly contended.
What go provides for abstractions around those queues are range and select. Range is a nice way to make simple concurrency cases look like traditional for loops (and is very rarely used in practice as simple concurrency cases are rare, at least for me).
Select is a nice addition to the language, yet the edge cases around channels make it very difficult to reason about without keeping http://dave.cheney.net/2013/04/30/curious-channels open.
> Have you actually written any Go? Using channels in Go is much nicer than using callbacks.
I have, for 2 years, full time. I have not used node. Have you ever used a language that supports concurrency besides Go? Go does not provide, as part of its concurrency abstractions, things that are deemed bare necessities by other languages such as timeouts, cancellation, supervision, lock free structures, etc.
The entirety of the Golang concurrency story can be wrapped up with 3 things a) the culture of the language prefers message passing concurrency b) the scheduler is a very good example of when simplicity gets you great performance in the main cases c) select as a language keyword is interesting.
By leaving the basics out you push all of that work to the application level, which is why I think the callbacks argument is a red herring. At the application level you are either going to encounter callbacks (as in the stdlib http handler) or blocking code (subject to races, deadlocks, etc).
What I was respond to was this specific sentiment: "With Go it doesn't matter if an operation is blocking or non blocking, that fact can totally be abstracted from the client code." My argument was that that is simply not true. Client code must know the difference between blocking and non-blocking code because it affects the flow of information around the program.
Please explain.
As for the go futures being bad, channels generally are a bad concurrent queue (contended, lack basic abstractions, etc) and using a single item queue as a future isn't in and of itself bad, but does mean you can't optimize for different usages that futures might have over message passing queues.
What is so hard about `require('lodash')`, or `require('moment')`?
> true concurrency
The trade-off is you get great concurrency without thinking about it. If you are working in Go, you have to start thinking about it as you go. You can always think about it as you go with Node.js if you want, but you don't have to and you will still get quite far.
> static typing
See http://flowtype.org or http://www.typescriptlang.org. With more advanced type systems than Go.
> With Go it doesn't matter if an operation is blocking or non blocking, that fact can totally be abstracted from the client code. Writing callbacks is tedious, promises are tedious, co routines with yield need plumbing and async isn't in the spec.
You can use ES7 async/await and promises today. That it "isn't in the spec" probably won't be for too much longer.
async function() {
const {data} = axios.get('foo')
const results = []
const arr = ['foo', 'bar', 'baz']
for (const i in arr) {
results[i] = await axis.get(arr[i])
}
return results
}This appears not to be true for lodash.
So upgrades are optional and not pulling the rug out from underneath the feet of devs.
Major releases allows the project to make corrections and avoid carrying around years of baggage.
But a standard library is a little different due to the sheer number of dependencies and level of coordination needed to upgrade them all. Go got it mostly right the first time, and the lack of churn is a major advantage.
async function() {
const {data} = axis.get('foo'),
arr = ['foo', 'bar', 'baz'];
return arr.map(ele => await axis.get(ele));
}
return arr.map(ele => await axis.get(ele));
return arr.map(async ele => await axis.get(ele));
return arr.map(ele => axis.get(ele));
That is, it returns an array of promises of values, not a promise of an array of values like erlich's function.
erlich's example is correct, with the replacement of for-in with for-of.
Edit:
Also, a solution using Array.map() is not impossible:
return Promise.all(arr.map(ele => axis.get(ele)));
This is not the endorsement you seem to think it is.
> My guess is that the Go community probably wouldn't exist if it wasn't for this constant form of aggressive propaganda/marketing. Node.js on the other hand doesn't need any marketing/propaganda; it just sells itself.
Your guess? This is pure emotion arguing here. I have rolled my eyes at the golang converts for literally years and recently I decided to give it a try and it truly sold itself, sans marketing. I also really like node.js, for what that's worth. This comparison sounds like the hurt feelings of the True Believer.
Different tools have different applications. Don't ever forget it.
> Of course if you're comparing a single instance of Go running on 4 CPU cores, it's going to be faster than a single instance of Node.js running on 1 CPU core.
I'm not intimately familiar with Amazon, so to me it's unclear what "2 medium ec2 instances" mean. It might mean, 2 m3.medium general purpose instances, in which case they actually expose only one cpu core. They also seem a terrible fit for an IO/CPU-bound work load?
While I can sympathise with the author that it'd be nice to spend as little as possible on a (micro) service, it strikes me as odd that they first tried going 2x medium -> 4x medium, rather than 2x medium -> 2x x.large (possibly a different intstance type, like c4 -- and run 8 (or whatever) node instances per vm.
It does sound like this is a typical area which golang would be a very good fit (Reminds me of: https://talks.golang.org/2013/oscon-dl.slide ) -- But it also strikes me as odd that rewriting the (apparently) simple service in Go ended up looking cheaper than a) scaling up (rather than horizontally), and b) rewriting the service in nodejs (but with an architecture more suited to the problem).
Node is not a speed demon. It's not as bad as Python, which can be insanely slow. But if you use Node, your server costs are always going to be some multiple of what they could be. (Also applies to RoR.)
Of course the usual argument is that servers are cheap and devs are expensive, so you'll be quicker to market with Node and that's worth the extra cost.
Which is fine until you need to run at speed and/or scale. If you start having to pile on the instances, you will be wasting money. If you're big enough to need hundreds of instances, you will be wasting a lot of money.
IME it's debatable whether Go can't also be faster for development. There's a learning bump at the start, but I found it easier to write code that just works, easier to write good tests, and easier to deal usefully with errors.
I am so tired of reading hacker news articles with comments that are so pushed down because of subjective opinionated voting that is so obviously skewed towards one personality type.
I have so many thoughts and opinions I want to contribute to these discussions but I have zero confidence it will fit within this type of environment that's been created.
It really saddens me to see some of the comments that get downvoted here.
At its worst, I've seen people get down voted for asking beginner level questions. The calibre of folk who post here can, by its nature, make commenting a nervous activity. Getting down voted for trying to expand your knowledge is both harmful and wrong.
If I see a comment down voted without the above properties, I up vote to neutralise whether I agree or not. If the comment contributes to debate or discussion, it should be valued. I know I value alternate perspectives.
The fact that down votes are silent makes things worse. Great, you've been down voted. Why? Ask why, down vote again. It's almost a recipe for gradual shaping to conformity of group opinion for anyone who wants to participate here. It can then come across as say these things, don't say these things and you'll be sailing.
I don't know, perhaps an explanatory comment should be encouraged so that real education happens. Otherwise its almost Pavlovian in nature, especially for the less confident and highly anxious.
Given the way HN is designed (comments are ranked, low scores get grayed out, user karma scores are visible) if people downvote to express disagreement it would worsen the experience of reading comments, and, I suspect, the overall conversation.
she
There are lots of bad things to say about Golang, but it's pretty wrong to suggest that PM2 makes it as easy to write performant systems as Go does. Go even has a really good race detector to help catch the few cases where you can get around channels.
The idea that a language cannot directly and transparently support scheduling across CPU cores is an antiquated one. Java is abandoning it. C# abandoned it long ago. Erlang showed long ago that we can do it reliably and with great performance.
Heck, even browser-level Javascript is moving away from this idea with async support, although everyone seems rather reluctant to make the plunge towards a stackless architecture.
> If you have a scalability problem with Node.js (or any language for that matter) and you can't find a simple solution that doesn't require rewriting your whole app, you're probably not qualified to manage this project at scale
I've got 13 years experience, 10 of which are as a distributed systems engineer. I've definitely hit the absolute edges of what off the shelf opensource software can manage, and more than once butted up against bugs and limitations of language runtimes. I'm extremely qualified to say that this is wrong.
Sometimes re-architecture and rewriting is the only way forward. That isn't necessarily a failure of the thing to be replaced. It's not even a failure of the architect if it is anticipated and managed correctly.
This is one of the most absurd and myopic statements I've seen in a while. Go has a vibrant community because many people chose it, for reasons that are entirely valid. Just as people chose Node.
You're posting with an enormously superior tone for someone that didn't even read the original post (newsflash: they were using a 4 host service cluster).
But there are times when it can be a pain. For instance, a node app I run at scale maintains an in-memory read cache that rarely gets updated. When we scale with processes, we end up having to duplicate the cache across every process. We could set up a helper process and have it manage the cache over a socket, or possibly write a c++ module that shares some memory but at that point it's no longer simple and/or there are more parts to potentially fail.
Yes, Redis can be the right tool. But, note how quickly we went from "I just want to run on multiple cores" to "I need to share data between my cluster workers so let's install Redis". It's just a total blowup of moving parts.
Great-grandparent argument reeks of elitism. I know how to deal with concurrency, parallelism, avoiding data races. Doesn't mean I need to waste my time implementing the primitives every project. To say that the language forcing you into multi-process architecture is a benefit, is a pitifully desperate remark. I wonder what single digit percentage of projects actually go into that territory? Maybe even less than 1%? Maybe with a runtime suitable for the task (e.g. Go or Java or hand-woven C++), less than 0.1%, because you do not run into pointless runtime bottlenecks to begin with?
I have implemented such projects, because we needed the workers to be geographically distributed, not because the language runtime fucked us over. This was a decision made in DESIGN not in DEFECT REMEDIATION.
I can think of a handful of "single-purposed" reasons to use Node on the server, such as DOM rendering (and other stuff that is only available in Javascript-land). But a serious project, no way. Not sure how people can stomach the inherent lack of safety in the language, the huge (relative to comparable substitutes) number of limitations in the runtime and the ecosystem churn.
Your mileage may vary but the JS ecosystem is mostly garbage from where I'm standing.
Want examples?
- Distributed worker queue
- Minimal web framework
- etc...
Ummm, both of them run in exactly one process, this is how operating systems work. Go can do it's n-m scheduling across multiple _threads_ sure, but threads are not processes (unless as an implementation detail of an OS). If you are running lots of instances of Node, that is a different thing.
Process to process communication has tremendous cost, although shared-memory does make that faster, but at the cost of complicating the interaction. Message-queue (ZeroMQ) based interaction also an option but, by definition, it cannot be as fast as languages that have concurrency support built-in.
Reasoning about concurrency is hard, and languages like Go (via CSP), Erlang and Scala (via Actors), Java (via Threads), Rust, and Clojure (via STM) take different approaches to make this less of a pain.
CSP and Actors are what seem the most 'natural' of solutions. And that's a major reason why Go and Erlang are frontrunners in the concurrency race.
Actors are pretty nice when it comes to distributed services, and Erlang/OTP is pretty much uncontested here.
Threads with manual locking are only useful when speed is really important, but this is somewhat of a moot point when it comes to concurrency. Both STM and Actors perform slower (starvation, message copying overhead).
Here [0] is a nice, in-depth discussion about some of the different approaches to concurrency.
[0] https://plus.google.com/109566665911385859313/posts/FAmNTExS...
Akka.io on JVM could pretty much do what Erlang can on BEAM, functionality wise. Although some people have raised concerns over it being a "pseudo" solution and that it is a no match for Erlang.
A single Node.js instance runs its business logic in a single process. A single Go instance can run its business logic in multiple processes. Of course if you're comparing a single instance of Go running on 4 CPU cores, it's going to be faster than a single instance of Node.js running on 1 CPU core.
But the thing the author fails to mention is that there are a lot of Node.js modules which help you easily scale across multiple CPU cores. E.g. PM2 https://www.npmjs.com/package/pm2
Had the author of the article actually understood the real cause of the problem, he would have saved himself and his company a lot of time.
Nodejs is still outpacing Golang according to Google Trends and yet I have never seen a single article about switching from X language to Node.js. My guess is that the Go community probably wouldn't exist if it wasn't for this constant form of aggressive propaganda/marketing. Node.js on the other hand doesn't need any marketing/propaganda; it just sells itself.
If you have a scalability problem with Node.js (or any language for that matter) and you can't find a simple solution that doesn't require rewriting your whole app, you're probably not qualified to manage this project at scale - Because scaling across multiple CPU cores is kids' play compared to scaling across multiple machines. Heck, my grandma could scale this thing across multiple CPU cores.
Personally I think Node.js is great because it encourages you to think about these problems sooner rather than later.