> wouldn't something like C be better?

Jim is an intermediate-level coder, by and far the average guy that you are going to get. He writes his IRC server in C. It performs acceptably and can be scaled horizontally. There are a few threading bugs and exploits (buffer overflows etc.).

Sally is an advanced coder, it took a year of recruitment to find her. She also writes her server in C. It's blazingly fast. Virtually nobody else understands how it works. She's a human, so it's still littered with the same types of bugs that Jim's server has.

Jack is at the same level as Jim. He starts off in Go 1.4. While his server is nowhere as fast as Sally's, its much faster than Jim's. Race conditions and exploits tend toward zero. Everyone on the team can approach the code and maintain it.

Go 1.6 is installed on prod and suddenly Jack's server is now negligibly slower than Sally's. Jim notices this and has to spend a few weeks on his to catch it up. Sally is stuck debugging a race condition that occurs once a month. Jack is adding more emoticons, more features and decommissioning servers in the cluster.

Edit: IRC is a simple problem and that begets a simple solution. While C may be significantly simpler than C++, Go requires far less cognitive effort: it is actually simpler than C.

"Simpler" obviously didn't happen here. They had to debug the garbage collector, which included looking at traces that went into the OS. They effectively debugged a Go and C mix.

I'm afraid the truth of these things is that if you try to squeeze maximum performance out of some of these more sophisticated languages, you have to be able to understand and debug the runtimes that come with them. Not that many people are up to that, which means the hurdle can be higher.

I'm not implying that C is objectively inferior. Quite the contrary: there are tools that suite a specific problem. Maybe Twitch did spend money tuning Go, maybe that was a waste. Point is: they go a free performance boost thanks to a language the prioritizes what you want to do instead of how you do it.

I'm a big believer in early performance optimization, premature if you will. Though, ultimately, what I can crack out in a day with C# is worth months of C. Iterate, even with languages. If, after profiling, you find that your expressive (as opposed to explicit) language is wanting it's time to iterate into a different language. Get something competent out of the door, spend more money on it only if you have to. Right now our socket library consumes less than 1% of the CPU (GC and all), until everything else catches up there is no benefit to getting any closer to the metal.

I find it hard to believe Sally is so advanced if she writes code that others have difficultly reading.

It shouldn't take a year to find someone who writes unmaintainable soup.

If you had me on an IRC server that had to service the numbers that Twitch is talking about; I would use every single server coding practice I am aware of. Someone else comes along. Maybe his first refactor would be to remove the buffer pooling. Maybe he'd change a data structure specifically designed to prevent false sharing, cutting throughput.

Good code doesn't always mean approachable code, writing a decent socket server in C assumes a ton of advanced knowledge.

If Sally had to write, say, a C logging library it would be a masterpiece of simplicity. These days your code has an audience, and those audiences can vary quite greatly.

I consider myself an "advanced coder", whatever that means and so I know that an IRC server is likely to be IO bound. I also know to avoid C, especially multithreaded C. Not because I would be unable to write correct multithreaded C, but because I probably don't have to to solve the problem adequately.

My experience means I know to pick my battles.

Disclaimer: despite all my praises for Go, I don't actually use it for real problems. Would I like to use Go? Definitely! Is there a problem that I'm trying to solve that needs Go? Certainly not. In many cases I'd objectively pick Rust over Go, but when it comes to threading primitives, Go is somewhat unmatched.

> pick my battles

Exactly my point. A good coder will choose the tool that expresses the solution correctly. C is a very good choice, it always will be, there are sometimes better choices. Ultimately it seems as though we are in agreement; cheers!

Go's threading primitives are not unmatched. You can do the same in Rust [1], but the gold standard in my eyes is STM in Haskell which allows you to block on arbitrary conditions.

[1] https://doc.rust-lang.org/std/macro.select!.html#examples

OT: why is an IRC server I/O bound? That really piqued my interest.

I've never made an IRC server but I share a similar feel for I/O (Network) being the limiting factor. Have one user write a message and then you need to send out potentially hundreds of thousands of messages. (At twitch scale)

The actual CPU computation going on per event is minimal (process maybe a few kb of text), and if we're only dealing with text, probably not memory (capacity or throughput) bound and certainly not disk bound.

Correct, fast, and maintainable C code is a very tall order. Good luck finding ANYONE who can do that consistently for a reasonable paycheck.

Huh, it's almost as if the foundation of all the devices we use on a daily basis are written in some language that apparently isn't possible to write correctly.

They are mostly reliable (though probably still full of security holes) despite C—made possible by the millions of person-hours thrown at the problem of maintaining large C codebases.

Back then, we had no choice. We can do better.

Essentially all but the most used code-paths with be bug riddled. This paper [1] on finding bugs in C compilers by fuzzing is an interesting case-study.

[1] http://lambda-the-ultimate.org/node/4241

Server programming isn't even that hard. You could do it in C all day. (Just have good sandboxing for when someone finds an exploit...)

Millions of people have been writing C for games, firmware, computer systems without Internet since the 80s on, and they didn't even get software updates. If your server crashes you get a coredump and you can update anytime you want.

s/isn't possible to write correctly/is difficult to write correctly cheaply in non-trivial applications/

and you pretty much have it. Building higher-level abstractions has historically been a good thing, unless your day job is punching in opcodes.

I was more poking fun at the hyperbole that it's not possible to hire anyone who writes C at a reasonable rate.

That's basically true.

Apparently it didn't have to be C or blazing fast.

I think this is why 'advanced' is used of 'better'. :)

I think that's true, but I don't like that it's true. What does that say about programming as a profession? Yuck.

Reading this causes me to experience déjà vu; years and years of reading stories and watching presentations about someone struggling with GC in the JVM. It's happening all over again with Go. The same 'discoveries', the same trade-offs, the same discussions about hardware resources, the same 'concurrent mark and sweep', the same 'more to do' conclusions. You could replace every occurrence of 'Go' with 'Java' and it would probably go undiscovered.

Maybe it's all worth it and this is how developers are supposed to spend their time, but it's no longer interesting to me.

It's because GC is a bad idea that's had 30 years of good research thrown after it. Advancing GC is building a faster horse instead of stepping back and building a car. It's time to move on, and I'm thrilled to see modern languages (Swift, Rust) abandoning it and focusing on building more intelligent compilers.

The goal shouldn't ever be to make the world's best GC, it should be to create the world's best way to elide lifetimes so that developers don't have to think about memory management. GC shouldn't be a goal, it's a technique for solving a problem, one of many that we should explore.

Seems like the opposite surely? We should be developing languages that more succinctly address the problems we humans are trying to solve not book keeping for the computer's hardware (that should be the computers job!).

I think you missed the bit where he said "more intelligent compiler". The compiler is the bit that does the bookkeeping, only in Rust (and evidently Swift--I haven't played with it much) it's done statically, ahead of time rather than at runtime.

That said, I think Go is a much more practical language than Rust for most problems. That said, I'm still very excited about Rust.

>I think you missed the bit where he said "more intelligent compiler"

Also known as "sufficiently smart compiler": http://c2.com/cgi/wiki?SufficientlySmartCompiler

These are different things. A sufficiently smart compiler is a hypothetical compiler that could theoretically optimize a high level language so that it could be faster than some low level language. This isn't what we're talking about here--we're talking about the concrete ability of the Rust compiler to preclude certain classes of errors.

Yeah, that's exactly what I was trying to say. Rust does all the bookkeeping at compile time, Swift keeps a lot of it at run-time although the compiler can easily optimize away a lot of lifecycle stuff too when it's in scope, so I assume it either does or will.

I agree that Rust likely does not have the be-all answer to automatic memory management, though what I love about it is that they're pushing the boundaries and getting people thinking differently about memory management.

> I agree that Rust likely does not have the be-all answer to automatic memory management, though what I love about it is that they're pushing the boundaries and getting people thinking differently about memory management.

Me too. I intend to use it for more of my hobby things, but Go is currently the best fit. Eventually I imagine Rust will pick up some decent GUI libraries or at least get decent editor support (vim-go is lightyears ahead of YCM+racer) and I'll be able to afford to justify using it more.

What he said is that the goal should be that developers need not handle memory manually. GC is one technique to achieve that goal, and the one that has been the most successful so far, however we should not equate automatic memory management and garbage collection: other techniques could offer an as good or even better experience if we took the time to explore, develop, and refine them.

GC is also required for persistent data structures which makes it a must have for languages where immutable data is a fundamental strategy for handling concurrency.

Thank you! Finally someone who talks sense.

> Reading this causes me to experience déjà vu; years and years of reading stories and watching presentations about someone struggling with GC in the JVM. It's happening all over again with Go.

It's because GC is an area full of tradeoffs, and despite popular belief, the HotSpot GC is really good. In fact, I honestly don't know of any way to improve on the HotSpot GC for general-purpose use (i.e. throughput/latency balancing). HotSpot has a generational, concurrent, compacting GC; allocation takes 4 or 5 instructions (really!); the compiler has SROA to aggressively optimize out allocations where unneeded.

Look up the Azul Systems pgc. That is more or less the holy grail, but it is patented out the wazoo, and is a commercial implementation only.

Also, the jrockit jvm (which was from BEA and was purchased by Oracle) is actually quite a bit faster than hotspot and easier to introspect (lookup jrockit mission control) than hotspot. I suspect eventually they'll merge however.

> Look up the Azul Systems pgc. That is more or less the holy grail, but it is patented out the wazoo, and is a commercial implementation only.

That's what I was alluding to in the parenthetical. According to the paper, C4 trades off a significant amount of throughput for reduced latency. That's what you want for many applications, and C4 is a great advance for those apps, but throughput is very important for most workloads, so HotSpot's GC ends up yielding a good balance.

Touche, I clearly didn't fully understand your original statement. You're entirely correct in this case.

So we finally reaching the point where batch and interactive jobs are clearly separated because the very different tradeoffs they need. http://www.winestockwebdesign.com/Essays/Eternal_Mainframe.h... indeed.

It's not perfect. Azul's C4 does a lot of work in read barriers, so code that looks intuitively like it should be fast can end up causing "read storms" that bog the code down.

C4 never pauses, and that's impressive. But there's no free lunch. The work the GC would do when the app is paused is sometimes being simply done by the app threads instead.

I heard the read storm problem has been solved by Shenandoah, a GC developed by Christine Flood (who was in the original GC G1 team at Sun in 2001). It is under the RedHat umbrella and should be merged in OpenJDK [1].

Shenandoah uses a forwarding pointer in each object, adding overhead but limiting the problem only to write barriers. Here is Christine commenting on Azul vs Shenandoah [2]

From the talk: average pause is 6-7ms, max is 15ms, and the talk is one year old.

She hints at further developments in a version 2 which would make it entirely pauseless.

She has made another talk at RedHat's DevNation conference a few days ago, but they just won't put the video online arg!

[1] http://openjdk.java.net/jeps/189

[2] https://youtu.be/4d9-FQZZoVA?t=13m11s

Does Java still have a word in every object allocation for locking it? Adding a 64-bit pointer sounds terribly inefficient.

Did you know Objective-C does locks and retain counting without allocating any extra fields in objects?

On hotspot: There are two bits in the header of every object. This is enough for an object that's never been used as a contended lock, CAS operations on the header can be used to handle the locking and that's that. As soon as you actually block on it, a 'real' lock is created (you can't get around the need for a list of threads to wake up as the lock is released) and the header grows to accomodate it. The process is called 'monitor inflation'. At a later date this might be cleaned up by a 'monitor deflation'.

There's a certain amount of work that has to be done for GC, and that work is going to be done somewhere. The question is just what trade-offs you want.

Don't want compacting? You'll pay for it in allocation.

Don't want pausing? You'll pay for it in application threads.

Precisely, and this is what is so often missed in these discussions. Most of the time, when you see claims of GC silver bullets, there's some hidden downside that's being papered over. Latency wins (i.e. "max pause time" or whatnot) tend to be throughput losses. Less copying results in more fragmentation. Value types can result in more copying, reducing performance over pointer indirections through nursery allocations. And so forth.

I don't know that anyone disputes this. The discussions I participate in don't deny this; they mostly talk about whether or not the tradeoffs result in a net gain (if you sacrifice a little from the minority of cases to gain the same amount in the majority of cases, you do indeed have a net gain).

> Value types can result in more copying, reducing performance over pointer indirections through nursery allocations

Having value types means you can pass by copy, but it also means you can allocate on the stack and pass by reference--in other words, you get performant passing without involving the GC.

Nearly all the work you allude to is done in other threads - which indeed consume machine resources (CPU cycles, memory bandwidth). If your application does not burn all cores/bandwidth then the GC work is all done on the idle/spare machine resources. At the limit though, indeed you'll have to slow down the Application so that GC can play catchup - and bulk/batch stop-the-world style GC's require less overhead than the background incremental GCs Cliff

> C4 never pauses

To my knowledge this is false. AFAIK while the C4 algorithm is pauseless the C4 implementation is not. It's just that the pauses are really short.

My understanding: C4 does not pause but the JVM still does for various other reasons and a part of the work on Zing has been forcing down those pause times too.

Sorta kinda all of the above. C4 the algo has no pauses, but individual threads stop to crawl their own stacks. i.e., threads stop doing mutator work, but only 1-by-1, and only for a very short partial self-stack crawl. C4 the impl I believe now has no pauses also. HotSpot the JVM has pauses, and yes much Zing was on forcing these pause times down. Cliff

Isn't it also because Java and/or having GC makes certain patterns easy although they should be hard? At least my limited experience with Java, from writing a system which dealt with millions of integers, is that Java really wants you to use ArrayList<Integer> instead of the GC-friendly int[].

The problem with Java is that most things are a pointer, which means the GC has to deal with it. Go on the other hand allows the user to specify which things should be a value and what should be a pointer, which significantly decreases stress on the GC.

C# has something called value types, and while this helps (and Java is working on implementing something similar for Java 10) it's not as flexible as Go, where users can decide this at whim instead of specifying it in the type.

Java doesn't "want" you to use ArrayList<Integer>, that's merely more convenient if you need a dynamically sized array and don't want to do the resize yourself.

But the JVM folks are adding support for ArrayList<int> to the language, with the efficiency you'd expect from it.

Value types would hopefully get a big help in terms of getting the JVM GC better once the SDK and popular libraries full utilize it, both by reducing memory pressure and making the heap more GC friendly.

Fun stuff I've been doing with the H2O project is basically using nearly-pure Java (some Unsafe) to hold onto numbers with better efficiency than e.g. int[], and giving out an easy-enough-to-use API for writing parallel & distributed code over an Array-like API. i.e., feels "almost like an array", and "for-loops" run at memory bandwidth speeds and also parallel and distributed. Cliff

And the actual data is stored in giant byte[] (hidden behind the API), so the GC costs are near zero. Cliff

Are the number of garbage objects generated by idiomatic Go and idiomatic Java comparable?

My guess is Go implementation will produce an order of magnitude less garbage.

In Go, an array of structs (= objects) is just one object.

In Java, an array of objects is array object itself plus one object for each value in the array. Except for elementary types, like bool, int, long, etc.

I completely agree and with so much tuning required in past frameworks for their GC it makes me wonder why more don't simply adopt the C++ / Rust models of resource management.

I remember way back when people said you couldn't use the JVM for real time applications because of the GC pauses but it's been improved significantly since then and now all the same topics are coming up with GO.

Because the C++/Rust way of memory management is better for some things, but worse for others. I've worked with several different projects during my career, and not once did we require manual memory management. A GC based language was simpler for us to use, and the few times we had problems with GC, they were possible to overcome by writing better code, as is the case with any language.

This is not to say that no project benefits heavily form C++/Rust. But I would argue that for many, GC is the best trade off.

I completely agree that explicit memory management (I wouldn't call it manual) in the C++/Rust way is a cognitive overhead you don't want for a great deal of the software work - perhaps most of it.

But there are definitely projects that require explicit memory management, and it's not just games and realtime software. Often high-performance backend code in Java and Go just end up using object pools instead of reallocating objects, just as the OP described.

With Go specifically we've seen the rise of fasthttp, which just adds completely manual memory management in the 90's C++ fashion. Want to create a new request object?

req := AcquireRequest() req.DoSomething() ReleaseRequest(req)

Compare to C++98:

Request* req = new Request(); req->DoSomething(); delete req;

And now you're back at the same manual memory management problem modern C++ and Rust are striving to solve.

I'd argue that one of the biggest differences between golang and java is not technical but cultural. That is, the golang idioms and thus the std libs are quicker to reach for things like object pools and other performance "hacks". Even the std http library uses an arena in golang.

Similarly, high performance Java libraries like the Disruptor, SBE or Chronicle look very much like C code.

Personally, that doesn't bother me, as it allows you to write your hot path and your non-optimized path in the same language with the same tooling.

says the guy who has split JVMs across processes for performance and contemplated doing it per core

For what, I would assume, is a minor portion of your total lines of code.

I'm not sure where you got manual memory management from. I was strictly referring to RAII. Manual memory management is such a pain but the C++ and Rust ways are very similar with RAII.

I usually refer to everything that isn't GC as 'manual'. RAII is an abstraction on top of manual management, you still need to decide what type of pointer/lifetime the allocated object should have, making it manually managed, IMHO.

RAII of course deals with more than just memory, but in a thread about GC I assumed it was memory management you referred to.

>Maybe it's all worth it and this is how developers are supposed to spend their time, but it's no longer interesting to me.

Unless you design and implement GCs yourself, it's not supposed to be interesting to you anyway. It's just something that will benefit users of the language, not something to excite them.

If it's not supposed to be interesting, why do so many that have found themselves running up against the limits of GC in their chosen language/platform/implementation end up writing epic blog posts that represent months or years of work and/or presenting their hard one solutions at conferences? There certainly seems to be a lot of people that end up having to be very interested in solving their GC problems once their systems grow to non-trivial size, and they all seem to be relearning and resolving the same set of problems.

>If it's not supposed to be interesting, why do so many that have found themselves running up against the limits of GC in their chosen language/platform/implementation end up writing epic blog posts that represent months or years of work and/or presenting their hard one solutions at conferences?

Because they care about improving actual, existing, languages, with actual, existing, ecosystems, not doing cutting edge academic memory management research.

>and they all seem to be relearning and resolving the same set of problems.

So like architects relearn and resolve the same problems, about building bridges, skyscrapers, condos etc -- instead of designing some new structures to replace them?

Go is Java 2.0, but from Google instead of Sun. At least the syntax is a little nicer and less boilerplated.

I sincerely hope Go does not go in that direction. The 'less is more' approach is so far very strong among the Go steering committee.

The cruel irony here is that simplicity was a foundational goal and major rallying point for Java. Here's a website from 1997 describing it: http://www.cafeaulait.org/course/week1/16.html

Ignoring the last part that obsesses over the glory of OOP, replacing Java with Go in that page is... pretty spookily familiar!

Maybe it's a bit too early to judge but IMO Go has not introduced any new language complexity since its public launch.

> I sincerely hope Go does not go in that direction. The 'less is more' approach is so far very strong among the Go steering committee.

There is no "less is more" approach in Go. It's more like you can't write something really complex in Go so people use it for trivial things like servers that do almost nothing aside from de-serializing JSON. Try write a large LOB app in pure Go or a fully featured CRM. And see if you can get away with "less is more" when you need to reason about complex business rules, data validation, complex routing, mapping RDBMS data to values, and what not. "less is more" is a mirage. Go short comings will show up pretty fast.

All you're really saying is that Go is not great for web apps. I admit, it is not. So is C. I would not write a CRM in either of these languages.

At codebeat (codebeat.co) we use Go for our backend - very CPU-heavy, complex static analysis workflows. Our frontend is in Rails which is not ideal but probably the best bang for the buck for an early stage startup. This is the beauty of having many tools to choose from.

> All you're really saying is that Go is not great for web apps. I admit, it is not. So is C.

C is 30 years old, so it has an excuse. Go has none. The fact that it's extremely difficult to write a classic, complex webapp in Go is a proof that this language has serious flaws.

It is just as easy as doing it in Rust or Swift - both being thoroughly modern languages. It's more about the lack of comprehensive frameworks than the language being somehow flawed. I used Go as a backend for mobile apps and it was OK but where it really shines is the kind of workload we do when analysing source code: where you need excellent performance and low memory footprint, all that while keeping the code readable.

> It's more like you can't write something really complex in Go

Have you recently checked out the bigger projects that are currently being written in Go? You'd be surprised...

> Try write a large LOB app in pure Go or a fully featured CRM.

Woah. Have you tried doing that in C, C++ or Rust? Has anybody? Every language has it's strengths and weaknesses. Sure it's possible to do so in them - but is it a good idea? Not necessarily. I'm not going to write a database engine in Python - but we have timeseries databases being written in Go.

> Go short comings will show up pretty fast.

Every language has shortcomings. Go's major thing seen as a shortcoming is the classic "lack of generics", which arguably is true to some extend - but not it's GC. The thing is - Go's strong points have become clear long before these shortcomings you're talking about. The entire ops-space jumped on it because it solved a few problems plaguing their tools: memory overhead, slowness, dependencies, hard to make portable. Pretty much every major new project related to infrastructure is created in Go.

One of the biggest attractions of Go is it's ability to create programs that perform a lot better than the same thing written in Ruby or Python, which then again allows developers to undertake more ambitious projects.

So does that mean Go will finally have generics somewhere around version 5?

Fingers crossed for the Hindley–Milner type system.

Even maybe algebraic data types? One can keep dreaming.

Then we might be in some alternate reality talking about how Twitch could never deliver a viable service because the developers kept creating segfaults. C is the last language I would choose in a race to a viable service.

>C is the last language I would choose in a race to a viable service.

And yet, the majority of services the world relies on everyday, from OSes and drivers, to Google search, NASA code, medical devices, etc are written in C/C++.

Too bad no one can deploy linux servers because of all the segfaults.

I'm flabbergasted that people are able to post to this thread at all, what with all the segfaults their C++ based browsers must be experiencing all the time.

I experienced quite a few kernel panics in the earlier days... Generally from poor drivers, but just the same.

At scale it may be more efficient to write the system in a higher level language (saving time) and then spending some time tuning only the slowest parts, instead of building everything from the start to be highly optimized, even the parts which may not need it (investing time where it may not produce results).

And the improvements to Go that they drove will help everyone.

I happen to prefer C, but I understand why they did it the way they did it.

Better in what way? Performance wise, perhaps but that's only one aspect of why someone might pick one language over another.

Even distilling better down to just the max throughput you can get for a solution in a language vs another is hard to do as a lot also depends on how the code ends up being written and how easy you want to be able to debug that solution. You can solve this stuff in C many ways with different performance characteristics.

In some ways it's accurate to think of Go as a more convenient version of C with modern facilities like automatic memory management, concurrency primitives and data structures (i.e. maps), with the minimal level of runtime scaffolding included to support them. Interop with C is very easy, and Go is miles away (stylistically) from some of the more esoteric and abstract languages that are used these days.

Interop with C is easy to code but deceptively expensive from a machine standpoint, due to correctness guarantees when you're switching from green threads to no-green-threads. It involves interacting with the go scheduler and possibly/probably blowing your cache and TLBs.

And the automatic memory management is great, but the above commenter was saying that if you're going to extreme lengths to work around the automatic memory management, maybe you needed a non-GC language in the first place.

Which ones though? How about Rust? To me, as a C programmer, Go looked like the closest one to my taste, but not enough or blitz to switch/consider.

C is not a very nice language for concurrent programming.

If you want to hire 300 people to write reliable software in a language they don't know yet, Go is a good choice. You might also have like half a dozen people who are so deep into Go that they do the stuff in this post.

>Go is a good choice

Until you ask them to do stuff with channels -- where Go offers 100s of subtle ways to shoot yourself in the foot.