joelgwebber

To be precise, Blink is moving to a GC for stability (including avoiding leaks), but I don't believe it's for security -- the renderer remains sandboxed because it's effectively impossible to secure such a huge pile of C++ code. This presentation (which assumes a lot of familiarity with the WebKit/Blink smart pointers) goes into some interesting detail: https://docs.google.com/presentation/d/1YtfurcyKFS0hxPOnC3U6...

It includes particularly intriguing bits like "You can remove all on-stack RefPtr<X>'s. This is the biggest reason why Oilpan performs better than the current reference counting." I don't know whether that always holds true -- as of the middle of last year, I heard that they'd gotten to the point where most things perform roughly at parity, some worse, and some better. Keep in mind that this is an opt-in system -- if you don't use the smart pointers the GC knows about, it will ignore them (IOW, it's not some crazy conservative beast like the C++ Boehm collector). Also, my understanding is that, the vast majority of the time, Oilpan only runs when the event loop goes idle, which makes perfect sense for a browser, and has an obvious correlate in a game's simulation/rendering loop. I think they only walk the stack looking for pointers in rare cases.

It's not hard to imagine a hybrid world where you opt-in to GC'd pointers, but are free to use different allocators for performance-sensitive bits. This smells a little like Rust, but without the need to satisfy the lifetime checker thing.

Thanks for the pointer on libco. I'll definitely have a look at that. I've not written much C++ (apart from Chrome and a few odds and ends while at Google) in a long time, so it's quite probable I've missed some significant improvements on that front.

C++ management can of course be workable with enough care. I worked on Chrome for a bit while at Google, and saw that it more or less holds together with enough reference counting and smart pointers. At the same time, it still requires a lot of care, and plenty of bugs have been caused by subtle mistakes in this kind of code (which is why Chrome uses a sandbox around the actual rendering engine, because it's far too complex to be trustworthy). But even Blink/Chrome is moving to a garbage collector (http://www.chromium.org/blink/blink-gc) for C++ objects because of all the memory management complexity.

What I'm hoping is that you can have a GC that allows you to avoid all these issues without having to be super-careful all the time, while mitigating the pause issue by reducing the garbage using pools and similar techniques. My hypothesis is that most of the little allocations that game engines perform are homogenous enough that moving them to pools will be fairly easy. And that this will be sufficient to avoid big pauses. But we'll see how it plays out in practice when I get some hard data on big scenes.

Finally, memory management isn't the only reason I'd prefer to avoid C++. I'm particularly sick of long compile times (they could really kill you on a big project like Chrome), and among other things I believe that Go's concurrency model will prove a big improvement over C threading.

Thanks for taking the time to comment, Jonathan.

This is why the approach I'm experimenting with is build something very much like a custom allocator in Go, for all values that are allocated in significant numbers. I'm hoping that this will take enough pressure off the GC that it will keep pauses below the threshold where they matter (see above for a caveat about needing a concurrent or incremental GC to avoid long, but less frequent pauses). For what it's worth, I'm not 100% certain that this approach will work well enough, but I'm hoping to get some data that we can use to debate this in more concrete terms.

If this does work well, awesome. If not... well, I'm still tinkering with Rust, but I found the type-parameter explosion off-putting enough that I decided to stick with Go for my first round of experiments. I'm curious how your experience with more limited (as I understand it, perhaps incorrectly) allocation annotations are working out in Jai. After all, I'm not dead set on using Go -- I just want to avoid writing C++ for hobby games if I can possibly avoid it :)

This was a fairly obscure bug lurking in the DOM/API bindings. They have to do a bunch of wacky stuff to deal with the interactions between the V8 GC and the ref-counted native API objects. In this case, the TypedArray wrapper objects were implemented in such a way that allocating them would often trigger a full mark/sweep (which also, IIUC, contained the logic necessary to free up reference cycles in native objects pinned by V8). My understanding is that all that stuff is much better now, and much less likely to lead to such pathological behavior.

Indeed. I'm hopeful that the concurrent GC will get it to the point where it can do a lot better than that "10ms out of every 50" promise in 1.4. If not, then well, I guess we will have learned that Go isn't great for games after all.

/me crosses fingers.

Nice work. Assuming my experiments convince me that writing games in Go is worthwhile, I'd love for the ecosystem to evolve to a point that it's trivially easy to get up and running. The fast iteration time is very nice for these competitions (now I just need a debugger).

I think it shouldn't be too much trouble to get to the point where we have a set of basic composable libraries for loading and rendering meshes and other graphics, as well as sound/input/etc. that doesn't require so much wiring. I haven't done the exercise yet, but I'm also hopeful that SWIG will provide good enough bindings to Bullet Physics, so that you won't stub your toe on "real physics" again :)

Yes, sort of. Not generating a large amount of garbage can still give the runtime an opportunity to reduce the amount of work done per collection pass. A naïve heuristic for determining when to run GC will give you the typical "sawtooth" pattern, because you generate garbage until hitting a fixed ceiling threshold, collect it all, then wash, rinse, repeat. But a game that's not completely pegging the CPU will have some idle time that can be used for small collections, so limiting the amount of garbage produced on each frame should put an upper bound on the amount of work done during each frame.

When I was helping Rovio port Angry Birds to the web a few years ago, we ran into serious frame hitches that were being triggered by GC pauses in Chrome. Two things fixed this -- the first was fixing a bug that caused it to run a full mark/sweep far too aggressively; but the second was when V8 committed an incremental collector (not concurrent, just able to spread the work out more by being able to run a partial mark/sweep and resume it later). After that, the GC pauses disappeared into tiny ~N00µs pauses that never impacted the game.

See my comment elsewhere on this thread, about techniques for limiting garbage in Go. I'm far from proving that this is sufficient for avoiding significant GC pauses, but I'm tentatively hopeful. I'll post more as I get more data on large scenes.

I did play with Rust a bit, and I do find a lot to like there. Unfortunately, I quickly found myself dealing with an overwhelming explosion of type parameters (both of the garden variety, and the 'lifetime' variety). Some of this may have been my own naïveté in the language, and some bad library design (the graphics library I was using ended up forcing me to pollute nearly every type with three or for type parameters). But that, coupled with my own Go experience, a slow-ish (though better than C++) compiler, and no better debugging support than Go, led me to stick with the latter for the time being.

See my earlier comment (and some bits of the original post). Go does indeed support arrays-of-structs, as well as taking pointers to the middle of arrays, and directly to struct fields. This gives you a lot more control over memory layout, and lets you avoid creating garbage if you're willing to put just a bit more work into it.

Here's my take on GC in games, FWIW. I'd be very leery of using a VM with a garbage collector for the entirety of a game. They can be fine (and are extremely common) in embedded scripting languages, but it can be far too difficult to control the size of outlier pauses when everything on the heap is subject to GC. As mentioned elsewhere in this thread, the JVM GC has had an enormous amount of effort put into it, and it's still an issue that poses problems for Minecraft, et al.

I'm far from proving this assertion yet, but I believe that Go's memory model allows for a middle way that will avoid big GC pauses. As I touch on briefly in the original post, you can use Go's C-like value types and pointers to field/elements to avoid generating garbage for large numbers of homogenous objects (e.g., by implementing a simple pool allocator), just like you'd do in C[++] to avoid heap fragmentation.

I hope to get more actual data on how this works as I expand my prototype, and will do follow up posts as I learn more.

Yes, this. I've often advocated for the importance of having a good debugger when developing any kind of UI, game, or simulation code. It can be really damned hard to make sense out of subtle bugs that are the result of accumulated state, using log messages alone. When I was writing games in C++, I routinely would just keep the game running in a debugger all the time, so I could break and inspect whenever anything got weird (which could often be difficult to reproduce). I'm writing server code in Go in my day job, and I agree that it's simply a different beast, and doesn't bother me as much.

With respect to the issue brought up elsewhere about debuggers and goroutines, I don't believe it would be as much of an issue with games. In practice, I only have a handful of fixed goroutines -- simulation, rendering, network, etc -- and am only debugging one of them at a time.

I'll be honest -- I hate working without a debugger. Go's gdb support was never great, and the team appears to have decided it's a fool's errand. On the other hand, the Delve project looks like it's serious about building a "real" debugger for Go (https://news.ycombinator.com/item?id=8595407). Supposedly it works on Linux, but I'm waiting on Mac support, which I understand is held up on Linux/Mach/Darwin differences.

In the meantime, I've gotten pretty good at printf() debugging (I worked on embedded systems in a past life, so it's a skill I've had to develop). I'm also considering adding some more structured log/trace/metrics stuff (perhaps exposed via a simple web UI) that would allow me to escape the "tyranny of the ever-scrolling console".

But in the end, I'll consider Go dead for game development (at least for me) if the debugger situation doesn't get fixed. I'm just betting that it will.

Damn, that was fast. Thought billing was enabled for that site, but I guess not. Fixed!

Edit: Or at least it should be. Might take a couple of minutes to clear up.

> I think that the original poster (the one you responded to originally) was talking about errors in the first two senses; things that you should/want to handle yourself. I guess the last thing should be handled with a panic?

Right -- I'm only saying that I want to handle the first runtime errors explicitly at the call-site. This is where I explicitly like Go's error style. Other approaches to this problem (e.g., pattern matching) have been discussed elsewhere on this thread, and that's fine for languages that want to go down this route, but the extra language complexity is a tradeoff. I can see coming down on either side of said tradeoff, but it's not cut-and-dried.

> It doesn't that you understand idiomatic uses of option-type [...]

Sorry, that came out wrong. Where the option-type/elvis-operator approach comes up, it seems, is when you need to dig a few levels deep through possibly-nil references, as in cromwellian's .getOrElse(foo).getOrElse(bar) example. I certainly understand how that can be useful, but in my experience it seems to come up most often when dealing with unvalidated inputs (I'm sure there are other cases I'm not thinking of; this is just my personal experience). Whenever possible, I tend to prefer having the inputs parsed, validated, and either accepted or rejected, by a validating parser of some kind. Then I really can just assume it won't segfault as I read through these chained methods/fields.

I don't want to pick nits too much here, but "[...] leading to the whole non-nil value being nil idiocy; it's mindblowing that they got this so wrong." So I paraphrased a bit :)

I understand that some may find the error handling a bit too verbose, but humbly submit that there's a legitimate tradeoff to be made in terms of language complexity vs. verbosity, and the Go team generally tends to fall on the side of simplicity over tersity. This works well for us, but not everyone will find it palatable. YMMV and all that.

The stack frame "problem" doesn't seem all that bad to me. It would be nice to have a built-in solution, but our own code for dealing with this is one tiny file someone banged out in a couple of hours, so I'd hardly call it more than a stumbling block. After watching the Java world deal indefinitely with untold design flaws in core libraries, I support keeping things simple at the outset wherever possible.

But hey, there are lots of choices out there, so I'm not telling anyone they must write their servers in Go. Just that it works well for us.

Sorry, I wasn't entirely clear -- I was referring to the original basis for the complaint about verbose error handling. I'm intentionally separating the handling of runtime errors (e.g., memcache request fail) from validation errors (garbage data causes a map to have a nil entry) and unexpected errors (whoops, that shouldn't have been nil).

For runtime errors, as I've stated above, I prefer explicit handling close to the error site. I think this is the right tradeoff for server code, though I'm less certain for client code.

For validation errors, I prefer, when possible, to have a parser that either succeeds or fails as a whole, and whose output I can then rely upon to be properly constructed.

For unexpected nils, I do like the option-type/monad approach, though in most cases (e.g., when writing Go, Java, C++, or Javascript) I just let them NPE/segfault.

Note that only one of these cases -- runtime errors -- results in a (result, err) pair in Go. The other two are just about result checking.

See my comment above for why I don't find this to be that great for server code. If each of those getOrElse() is an actual potential runtime failure (as opposed to just something that could in theory be nil, but you know won't be, by construction), then it's something you're going to need to deal with explicitly.

E.g., think of each of those method calls as being a call to some external system (filesystem, memcache, database, etc) that might fail, even if the code's correct. Throwing an exception is usually a pretty bad strategy for those cases if you want to able to sort out what went wrong later.

My personal experience at a startup with a large amount of Go code running its frontend and backend servers. YMMV.

First off, I also find the interface-nil thing to be a frustrating edge-case. I also know that this design was the result of some difficult tradeoffs, as such things often are. We can argue about whether they made the right tradeoff, but I don't think it's fair to refer to it as a "mindblowingly wrong idiocy".

Second, on error handling. Having now written a large amount of server code in Go, I find that I strongly support their approach, even when I find it a bit verbose. Here's how I find it actually plays out in practice:

    // You write something like this:
    result, err := getResults()
    if err != nil {
      return err
    }

    // Then you reuse err for the next call
    result, err := getResults()
    if err != nil {
      return err
    }

    stats, err := computeStats(result)
    if err != nil {
      return err
    }

Eventually, you realize that `return err` just isn't enough most of the time, because it's impossible to make sense out of your logs. So we added a simple "error chaining" function that allows you to pass context when you return the error, which makes the logs much clearer than just a single error message, or raw stack trace. And of course it is often the case that you want to do more logging in your error blocks, even as you ignore the error and move on (e.g., "spurious error reading memcache entry; falling back to the slow thing").

The practical reality of writing good server code is that exceptions which get caught way up the stack are inscrutable in your logs (at least that was our experience), so it makes sense to know up-front exactly where failures can happen, and to be forced to think about them the first time you write your code. Yes, it can be a bit verbose, but we've found the tradeoff to be net positive.

I might be able to do that more effectively if you'd point out what browser you're running.

Good point. Apple helpfully reports the rather confusing version string "Apple clang version 4.1 (tags/Apple/clang-421.11.65) (based on LLVM 3.1svn)" :P

Ditto for gcc-4.8. Roughly the same (ever so slightly worse) performance.

FWIW, I just ran it through Clang 4.1 (the standard on XCode these days), and got roughly identical performance. I'm still compiling gcc 4.8 just to be sure, but it seems unlikely to make much of a difference.

What phone browser are you using? It seems ok on my Nexus 4 and mobile Safari, but I may need to fix the viewport or some such thing.

It looks like Farseer is based on Box2D, but extended, so it makes a kind of sense. I suspect the .NET VM performance will be somewhere around what the JVM gets, so it shouldn't be too much of a surprise.

I must admit to being unfamiliar with Farseer, but if you feel like submitting a pull request, I'll figure out a way to run it on my Mac (I presume the "platform" you're referring to is .NET, or perhaps WinMo).

For the record, the size of Postgres is at least an order-of-magnitude off from what counts as a "large" C++ program, at least as far as Google is (and many others are) concerned.

Try compiling Chrome (or just WebKit) sometime. It can take well over an hour on a beefy workstation. Incremental compiles are, of course, a lot faster than this, but a long way from "interactive" by any reasonable definition.

But wait, there's more! That bizarre crash or link error you're getting on an incremental compile? Yeah, it's the result of some screwy preprocessor problem that no one's taken the time to debug. You could try and get to the bottom of it yourself, but that would take somewhere between an hour and a couple of days. So instead you suck it up, clean the build output, start up a compile, and go grab lunch. This is the day-to-day reality for WebKit developers, and I don't accept that we can't ultimately do a lot better.

Large server-side C++ programs at Google suffer the same problem, because there's an enormous amount of shared code to compile. It's not as much of a dependency and script/preprocessor rat's nest as WebKit, but it can still get really slow, and you still occasionally have to give up and clean/rebuild when things go inscrutably wrong.

Any assertion that C++ header hell is "good enough" and/or "not that bad" flies in the face of this reality.

Not really.

V8 was simply a drop-in replacement for existing Javascript VM's, so there was no "adoption" curve, per se. It ran something like 40x faster than the one in WebKit when it was released, so everyone just said "Awesome, thumbs up! Stick it in there."

Google is spending some developer relations resources promoting Dart, probably because it's the kind of project that requires more public buy-in (including a VM that ultimately needs WebKit hooks, which is not solely controlled by Google). But it still has to surmount the same hurdles to internal adoption that other tools and languages do. I'm no longer privy to the details, but I'm fairly certain that there are just a few internal teams trying out Dart right now, to help it get its "sea legs".

And that's really the way it should be. These things don't happen overnight, and edicts from on high that a particular team will use a new technology don't usually work out well.

I think you're fundamentally misunderstanding the way these kinds of things happen at Google. Most projects of this nature have to work to get internal adoption just like any external toolchain. There's almost never an "edict from on high" that a project will use a particular technology (except to the extent that most code in production has to be written in a set of approved languages; a language created by Google itself still has to surmount this hurdle).

Pike, along with other Go team members, have been heavily involved in solving very large-scale problems (e.g., http://static.googleusercontent.com/external_content/untrust...) for a long time now, so they are solving the pain they've personally experienced. But other teams at Google will treat Go just like developers in the world at large do -- with skepticism, until it's proven itself. It's doing so now, but it will take some time before you see heavy adoption, which is the way it should be with any new system of this magnitude.

Try thinking of it in terms of Android intents/activities. The vast majority of the time, it's handled automatically for you (where "it" is whatever you were trying to do), but when you want the option to choose, it's there.

In some cases (e.g., sharing) it's clear that there's never a single good option -- the choice should probably always exist. There are plenty of others (e.g., image editing) where a site might want to have a specific default, but allow for choice where appropriate.

The alternative, I'm afraid, is what we have now -- you only get the options (usually just one) that an app's developer happened to find time to integrate by hand, usually poorly.