News for hackers. Stuff that matters.
Re:Ditching Go for Node.js by Thaxll on Thursday November 23, 2017 @11:47AM (#15766198)
OP is complaining about Goroutine stack size at 4kb per connection, his test shows that Node8 is taking up to 150MB of memory with 5k users, 4kb*5k = 20MB for Go memory, I don't understand how Nodejs can take less memory than Go, and without real numbers / test I'm pretty sure he's doing something wrong somewhere.

From my experience on some large prod deployment, Nodejs app takes much more memory and CPU vs Go app for doing similar work.

Re:Ditching Go for Node.js by sebcat on Thursday November 23, 2017 @12:44PM (#15766627)
I count three goroutines per connection: One for ChatHandler.Loop (when invoked from ChatService), one for ChatHandler.socketReaderLoop and one for socketWriterLoop.

Each ChatHandler has three (buffered) channels: outgoingInfo.channel, sockChannel, readErrorChannel (not buffered)

There are some things that could cause bottle necks that are not related to goroutines and channels: Every published message calls GroupInfoManager.GetUsers to get a list of all the users in a group. That can be expensive (I don't know if it is or isn't). And then for every user in a channel, their userOutGoingInfo is retrieved.

I would suggest profiling before switching languages.

EDIT: changed "benchmarking" to "profiling" EDIT2: changed "are likely to" to "could" (... cause bottle necks) since I don't know

Re:Ditching Go for Node.js by bryanlarsen on Thursday November 23, 2017 @11:52AM (#15766246)
It's 2 per connection, so that's 40MB of RAM. The OP also said that was minor compared to the channel overhead -- it seems the number of channels was exponential to the number of users in a room.
Re:Ditching Go for Node.js by todd8 on Thursday November 23, 2017 @12:53PM (#15766690)
A fully connected graph of n users contains n*(n-1) links if users don't connect to themselves (I would describe this as polynomial growth in the number of channels, a lot better than exponential). A chat broker that acts as a switchboard between users could, I suppose, reduce this to a linear relationship between users and channels.
Re:Ditching Go for Node.js by amelius on Thursday November 23, 2017 @02:07PM (#15767157)
You have to divide that by 2. It's "half the matrix".
Re:Ditching Go for Node.js by todd8 on Saturday November 25, 2017 @09:48AM (#15776147)
Thanks, my mistake. You're right, there are n(n-1)/2 arcs in the fully connected graph (K_n).
Re:Ditching Go for Node.js by adamweston on Thursday November 23, 2017 @01:55PM (#15767096)
Like a bus?
Re:Ditching Go for Node.js by Thaxll on Thursday November 23, 2017 @12:00PM (#15766327)
Goroutine takes 2kb, so 4kb for read / write per conection.
Re:Ditching Go for Node.js by sebcat on Thursday November 23, 2017 @01:39PM (#15766997)
> it seems the number of channels was exponential to the number of users in a room

I do not see this in the code linked to from the article.

Re:Ditching Go for Node.js by anonymoushn on Thursday November 23, 2017 @11:53AM (#15766259)
Seems like it should be linear??
Re:Ditching Go for Node.js by iends on Thursday November 23, 2017 @11:59AM (#15766311)
Hopefully it's linear, or it's implemented wrong.
Re:Ditching Go for Node.js by bryanlarsen on Thursday November 23, 2017 @12:23PM (#15766503)
Hopefully, but all he said was "fanout" which could be either.

He knows it's implemented wrong, that's the point of the article: that Go doesn't make it easy for him to implement it right. The only question is the degree of wrongness. :)

Re:Ditching Go for Node.js by weberc2 on Thursday November 23, 2017 @01:47PM (#15767059)
I don't think this criticism holds water since he opted for JavaScript. If he was having problems with goroutines and channels, he could have picked a Node-like architecture (single-threaded, callback-driven) and still likely enjoyed better performance (and optimization tooling!) than with Node. If his issue was Go's type system, then he shouldn't have chosen a language with a strictly worse type system. In other words, in the worst case, some scenarios may require you to drop into `interface{}` in Go, which has the same safety guarantees as any type in JavaScript. This way only ~3% of your code is type-unsafe instead of JavaScript's 100%.

While I think there are lots of valid criticisms of Go (concurrency correctness is still hard and its type system is not very good for certain tasks), none of these are reasons to switch to JavaScript, as the author did. If he was struggling with goroutines and channels, he could still elect for a Node-like architecture (even making it single-threaded by setting GOMAXPROCS=1). If he needed generics and unions (as he cited), he switched to a language without them and without any static types at all (or a single static type, if you will); in Go he could have downgraded to `interface{}` which is analogous to JavaScript's one static type, and he would have only gave up type safety in the bits of code where Go's type system was lacking instead of everywhere.

I don't want to give the impression of overselling Go here; it's just that for the cited criteria, Go is strictly better than JavaScript.

Re:Ditching Go for Node.js by always_good on Thursday November 23, 2017 @04:24PM (#15767890)
In Javascript you have the option to gradual type into Typescript which offers much better static typing than Go. So I wouldn't be too dismissive.

Also, the "just write Node-like code in Go" isn't a solution at all. You're back to fitting a square peg in a round hole.

Re:Ditching Go for Node.js by weberc2 on Thursday November 23, 2017 @06:27PM (#15768533)
> In Javascript you have the option to gradual type into Typescript which offers much better static typing than Go. So I wouldn't be too dismissive.

It's not reasonable to switch from a language that gives you type safety in ~95% of cases to a language that gives you type safety in 0% of cases but which provides an easier transition to a type-safe language. This is why I'm dismissive.

> Also, the "just write Node-like code in Go" isn't a solution at all. You're back to fitting a square peg in a round hole.

You're wrong here. I read and write lots of Go code, and it's perfectly idiomatic to write single-threaded, asynchronous code. In fact, I'd probably do just this for his application, modulo a thin compatibility layer to deal with the fact that net/http spins up a goroutine for each request.

Re:Ditching Go for Node.js by adamweston on Thursday November 23, 2017 @01:58PM (#15767111)
Node makes the Node-like architecture much easier than it would be in Go, sort of by definition. Also, wouldn't Node's JIT fare much better in the face of dynamic typing than using reflection (interface {}) everywhere?
Re:Ditching Go for Node.js by weberc2 on Thursday November 23, 2017 @09:22PM (#15769063)
No, there's nothing special about it, and I probably shouldn't have used that term because it might convey that it is particular to Node. Event loop, callback based programming is easily implemented in most languages.
Re:Ditching Go for Node.js by Vendan on Thursday November 23, 2017 @02:16PM (#15767217)
interface{} is not reflection, and using both reflection and excessive amounts of interface{} are not a good idea in Go, and frankly, I don't see anything in this code that would require using either
Re:Ditching Go for Node.js by weberc2 on Thursday November 23, 2017 @09:20PM (#15769059)
To be clear, my point is that interface{} is the worst case in Go and the only case in JS.
Re:Ditching Go for Node.js by bryanlarsen on Thursday November 23, 2017 @01:52PM (#15767084)
Sure, if you don't mind writing your own websocket and pub/sub libraries.
Re:Ditching Go for Node.js by zsombor on Thursday November 23, 2017 @03:59PM (#15767749)
Both Go and JS are async and callback driven. The difference is that with JS those nasty callbacks are painfully explicit, whereas it is all hidden in Go. Node's callstack is a goroutine, except that the latter can also be preempted. If you have 100000 requests in progress in Node, it will consume at least as much memory as a well written Go program would.
Re:Ditching Go for Node.js by iends on Thursday November 23, 2017 @01:23PM (#15766879)
It being implemented incorrectly is not a property of the Go language, though.
Re:Ditching Go for Node.js by KirinDave on Thursday November 23, 2017 @04:29PM (#15767918)
It is a property of the Go language that the most recommended and explained concurrency primitives are slower than even 20 year old implementations.
Re:Ditching Go for Node.js by JepZ on Thursday November 23, 2017 @12:04PM (#15766349)
And I thought the memory usage of channels depends on the buffer size.
Re:Ditching Go for Node.js by weberc2 on Thursday November 23, 2017 @01:48PM (#15767066)
This is correct.
Re:Ditching Go for Node.js by tootie on Thursday November 23, 2017 @04:56PM (#15768047)
Java or C# solved all of this like 10 years ago but people like making this hard on themselves.