News for hackers. Stuff that matters.
Re:Random Acts of Optimization by gambiting on Thursday October 29, 2015 @06:48PM (#10474958)
I work on a major AAA title due to be released in a few months - our compile times are 20-25 minutes for Windows build(Debug/Release/Final), 10-15 minutes for Xbox One/PS4(consoles link much faster), and for Retail builds, with link time optimization, the build takes anywhere between an hour, hour and a half. That's on a machine with a 6-core, 12 threaded intel Xeon(we use distributed build anyway), 64GB of ram and the source code sitting on a 1TB SSD. The baking of data takes another 10-30 minutes depending on config. Anything that improves our workflow is very,very,very welcome.
Re:Random Acts of Optimization by roel_v on Thursday October 29, 2015 @08:08PM (#10475396)
I probably misunderstand, but if this is a critical part in your workflow, why wouldn't you

- put 'data baking' (I presume that means things like packing images into blobs etc?) on a separate server - put another few computers into your 'compile farm' if you're using distributed builds already anyway

How much of, let's say, an hour is spend on compiling, linking and packaging? Compiling is 'trivially' (for some values of that word) parallelizable.

Re:Random Acts of Optimization by bhrgunatha on Thursday October 29, 2015 @10:31PM (#10475821)
> Compiling is 'trivially' (for some values of that word) parallelizable.

Really? I've always heard that it's virtually impossible to parallelize. Or do you mean a particular step in the pipeline can be parallelised - like parsing separate source files?

Re:Random Acts of Optimization by roel_v on Friday October 30, 2015 @12:02AM (#10476091)
The actual compilation of each "translation unit" (roughly, ".cpp file"), so the preprocessing, lexing, parsing, AST generation as well as the first-pass code generation, can be done in parallel. A project with 1000 .cpp files would (theoretically) need only as much time for that phase as the longest of the 1000 would take (plus time for the overhead of transferring files to build machines).

Of course after that there is global optimization and linking, maybe those take the majority of the time on the GP's case, which is why I was asking. In my experience, even for optimizing release builds, those phases are just a small part though - 25% or so for my projects, as a high-end guesstimate? But as I said, maybe it's very different for others.

Products like Incredibuild do just that.

(of course it's not 'trivial' as in 'I'll set it up over lunch', there are many many details to work out, which is why I said 'for some values of that word' - i.e., the engineer's meaning of 'trivial').

Re:Random Acts of Optimization by gambiting on Friday October 30, 2015 @05:19AM (#10476830)
We actually use Incredibuild(and SM-DBS as well). Those times are distributed build times. Compiling on a single machine would take at least an hour.

Also - we do bake data on a separate server too, I just prefer to do it myself because then I have several configs(one small one with a test map, one bigger one with the main story etc), and they all work with my changes.

I would say that if I got latest right now, it would take me at least an hour to start the game for the first time on either console(game server is part of Win64 solution so I need to build that first).

Re:Random Acts of Optimization by bhrgunatha on Friday October 30, 2015 @02:37AM (#10476408)
I suppose I'm thinking of compilers in general, rather than C/C++ specifically though.

Scala (to pick a random example) is quite hairy to compile efficiently from what I've heard.

Re:Random Acts of Optimization by pandaman on Friday October 30, 2015 @12:18AM (#10476137)
In C++ (and C for that matter) separate source files are separate compilation units. So it is trivial to parallelize. The compilation process produces object code, which, generally, has to be linked into an executable to be useful and that is usually not a parallelized task even though nothing prevents you from writing a parallel linker in theory. Unless LTO (link-time optimization) is enabled, the bulk of the linker's time is in I/O so making it parallel would not produce any performance gains till recently. Nowadays we have SSDs that can benefit from deeply queued I/O and, probably, somebody will write a parallel linker eventually.
Re:Random Acts of Optimization by oso2k on Thursday October 29, 2015 @07:27PM (#10475202)
Thank you for proving my point gambiting. @forrestthewoods, asset baking is already 50-200% of the gambiting's non-Retail workflow build time. This is before PGO. Ask him, if had to choose between reducing his build time 10% (without a game performance penalty), or, increasing his build time 1000% (with a game performance improvement of 15%), I think he'll quickly choose the first option. Debug time/feedback is much more valuable (usually) [0].

[0] http://prog21.dadgum.com/47.html

Re:Random Acts of Optimization by forrestthewoods on Thursday October 29, 2015 @08:31PM (#10475473)
If you offered me mildly faster iteration OR mildly framerate at 10x compile I'd also choose the former. Iteration speed is king.

But it's not either/or. If you offered me Debug/Release/Retail(LTCG)/SuperRetail(GPO) I'd love to have that fourth option. It doesn't have to come at the expensive of any other build configurations. Even if PGO takes 10 hours to compile. It's part of the overnight build. Great! There's no shortage of AAA games that have an overnight 12 hour process to build lightmaps for a single level. And that's using a build farm.

But so far I've never heard a major, or even minor, PGO success story so it's all kinda moot. :)

Re:Random Acts of Optimization by adrianN on Friday October 30, 2015 @03:08AM (#10476476)
Count yourself lucky. It takes me 90 minutes just to compile the module tests for one module.