News for hackers. Stuff that matters.
Re:Memory and C++ Debugging at Electronic Arts [video] by danjayh on Thursday October 08, 2015 @01:22AM (#10351042)
OK. Maybe I just don't understand game development very well, but I don't get why some of the problems they had in the post-MMU era even existed.

1) Memory Fragmentation - why? This seems like it would only be a problem if they ran their entire game in the same virtual address space.

2) Subsystem A corrupting Subsystem B's RAM - once again, wouldn't one run the sim code in one address space, the rendering in another, etc. and have strategically shared pages for them to transfer data back and forth?

It really seems to me like the idea is 'yeah, we had virtual memory, and it was great, but we didn't really use it'... Granted, they put guard pages on some allocations, but that's akin to using supercomputer to solve your family budget. Lots of power available, but using it in a trivial way.

I acknowledge that there's some cost to a context switch, and that the 2005 generation used PowerPC with its fairly cache-inefficient hashed pagetable (poor locality), but the benefits of having a hard wall in memory between major system functions is immeasurable (I'm guessing that on PPC, they just set up a few Block Address Translation registers and called it 'good enough').

Then again, I come from a safety critical background .. we make aerospace code, so it's developed using the methods described in the article about 'the right stuff' that was up a day or two ago. We're really fond of having really hard walls between every resource we can (memory, CPU time, I/O bandwidth, FS bandwidth ... you name it) for major system functions. I realize that in game programming most of those walls aren't practical (due to performance impact) or necessary, but the memory one is fairly cheap and incredibly helpful.

PS - if you're wondering, we prevent memory leak problems by just disabling dynamic allocation post-boot, which happens on the ground. Our stack utilization is also required to be statically analyzable, which means that you can't do things like recursion. These limitations make the design of some algorithms extremely painful, but what other software have you ever heard of that can truthfully claim 0 memory leaks, guaranteed. We also do 100% code coverage (at the machine language level), 100% decision coverage, static and dynamic worst-case timing analysis, etc. etc. etc. Look up 'DO-178B' or 'DO-178C' if you're curious about what's done and why safety critical stuff is insanely expensive. More recently we're throwing in fuzzing for good measure, but fuzzing doesn't turn up a whole lot in extremely thoroughly tested code that's been reviewed line-by-line by dozens of engineers (and is subject to strict design limitations).

Re:Memory and C++ Debugging at Electronic Arts [video] by pandaman on Thursday October 08, 2015 @02:33AM (#10351219)
Here is all you need to know to understand game development. It's just two things.

Firstly, on a home console there are just two playable frame rates: 30 fps and 60 fps. This means you can only have frames either less than ~16.6 ms or ~33.3ms long. The next step is 20 fps and if you were even allowed to ship a game like that it would not make any money so you would likely get fired anyway.

You will do very unsafe things to stuff as much as possible into a game's frame because your competition is already doing it and your designers demand it. There is only one thing you could possibly sacrifice a little bit of performance for - development time.

Thus there is this second thing: a game has to ship before you run out of money. The most common production time is about 18-24 months. This is for millions LOCs of source code and terabytes of source assets (textures, models, video, sound etc) written, often from scratch, and tested.

How long does it take to ship software for a new plane in your business? I'd be terrified to even approach a plane that had shipped in 1.5 years, least fly in it.

Re:Memory and C++ Debugging at Electronic Arts [video] by bluecalm on Thursday October 08, 2015 @04:15AM (#10351459)
>>Firstly, on a home console there are just two playable frame rates: 30 fps and 60 fps.

This is probably very basic but I am going to ask anyway: why are 40fps, 45fps or 47fps not possible?

Re:Memory and C++ Debugging at Electronic Arts [video] by MaulingMonkey on Thursday October 08, 2015 @04:32AM (#10351509)
Because your monitor is (probably) refreshing at 60hz. Your frames will display for some integer number of monitor refreshes - at 1 you'll get 60fps, at 2 you'll get 30fps.

Over a second, it's of course entirely possible to get something in-between, where some frames are displayed for 1 refresh, and others for 2 refreshes. Inconsistently waffling between these is going to be more jarring than sticking to one.

Recently, you have things like nVidia's... gsync? Which make the monitor refresh rate variable with compatible monitors, to match the game framerate, making 45fps entirely possible, and reducing the jarring from varying frame lengths. But this recent, rare, and likely PC and/or GPU vendor specific still.

Re:Memory and C++ Debugging at Electronic Arts [video] by pandaman on Thursday October 08, 2015 @04:30AM (#10351502)
The video out is fixed at 60hz so, at constant framerate, you can only output every frame, every two frames, every three frames etc. In other words you framerate is a factor of 60. You could tripple buffer and do variable rate so an average could be something like 40, but it would cost you 8M and variable framerate is perceived worse than constant. If you cannot hit 60 you better spend more time on getting game prettier at 30 than waste memory on a tripple buffer and get a stuttering game as a result. On current gen memory is not as much an issue but the variable framerate is still not perceived well.
Re:Memory and C++ Debugging at Electronic Arts [video] by NickPollard on Thursday October 08, 2015 @04:27AM (#10351492)
Because the display is normally running at 60hz, so if you don't want tearing (where the frame to be displayed changes halfway through monitor refresh, leading to a sharp tear across the screen), you need to run at an integer number of refreshes per frame (known as vsync). 1 = 60fps, 2 = 30fps, 3 = 20fps.
Re:Memory and C++ Debugging at Electronic Arts [video] by jharsman on Monday October 12, 2015 @04:00AM (#10373111)
Traditionally TVs only support 60 Hz refresh rates (or 50 Hz for older PAL sets), so you either render a new frame for each frame the TV can refresh, or you display a frame for two TV refreshes.

Thi sisn't strictly true any more, since many TVs now support 72 Hz (to be able to display 24 fps content like film), but my guess is that doesn't have wide enough support to rely on.

Re:Memory and C++ Debugging at Electronic Arts [video] by corysama on Thursday October 08, 2015 @03:34AM (#10351359)
1) Virtual memory only helps you to the granularity of a page. So, 4k at minimum. Meanwhile, a PS3 has exactly 256 megs of memory available to the CPU and we are highly motivated to use every last goddamned byte of it because if we don't, our competitors will. Meanwhile 2, the vast majority of allocations are under 512 bytes. In practice, even that subset is mostly <64 bytes. In that situation, fragmentation within a single page becomes serious business. You can't afford to have 10% of a given page be 9-15 bytes gaps between allocations. That could add up to 15-25 megs out of 256 wasted! When you get really tight, you can't even afford a heap header for each allocation --thus the specialized small block allocators.

2) A lot could be done better here. 4 gigs is not a ton of address space to protect different systems from each other. But, if you assume no system is going to ever need more that 64 megs, then you could divide memory into 64 spaces of 64 megs each and never have to reuse memory between different systems. I haven't heard of anyone doing this. You'll still get corruption from garbage data ending up in pointers. But, virtual memory segregation would cut out a large class of problems.

Static allocation post-boot is not an option in games. You can be very strict and structured, but the needs of different situations are still too varied. Sometimes you need 90% UI, sometimes you need 10 gameplay regions, sometimes you need 100% devoted to a single, small room for a boss fight.

Similar to fuzzing, I've worked on games that user controller-monkeys. An AI "player" was set up to artificially progress through the game while spamming the system with random actions. We would set all of our machines to monkey-mode each night before going home. In the morning, we would have a fresh set of obscure crashes to investigate.

Re:Memory and C++ Debugging at Electronic Arts [video] by danjayh on Thursday October 08, 2015 @11:14PM (#10357949)
you actually get 4 gigs per process, so you could have 64 spaces of 4 gigs each, with the 256mb of physical ram mapped into those spaces as needed.
Re:Memory and C++ Debugging at Electronic Arts [video] by corysama on Friday October 09, 2015 @04:30PM (#10363082)
No app-space multiprocessing on consoles. The OS is extremely minimal. In the PS2 days, there was no OS. Just the BIOS and a few standard libraries to statically link.
Re:Memory and C++ Debugging at Electronic Arts [video] by panic on Thursday October 08, 2015 @01:29AM (#10351063)
Since the VM system works at the granularity of pages, using N entirely different regions wastes up to $PAGE_SIZE * N bytes of memory just in the free space at the end of the region. You also make your fragmentation problem worse, since you can't place an allocation from one system within the free space of another.
Re:Memory and C++ Debugging at Electronic Arts [video] by danjayh on Thursday October 08, 2015 @01:45AM (#10351104)
Not suggesting that everything go in separate regions, but let's say they have N major functions (where N < 10) .. maybe physics, AI, render, network, etc. that would each run in their own address space. If N were even, say, 20 this still only comes out to 20 * 4KB (PPC page size) = 80KB, which I would classify as trivial.
Re:Memory and C++ Debugging at Electronic Arts [video] by MaulingMonkey on Thursday October 08, 2015 @03:59AM (#10351412)
You have to manage memory for things like textures, models, and maps - each of which are generally going to be way larger individually than 80KB.

Here's an example of a pathological - but entirely reasonable to encounter - case:

You allocate several textures of various sizes. Your larger 3MB texture blocks might end up seperated by smaller allocations for, say, icons. Or other 3MB textures that can be reused on the next level (to reduce load times). Long story short, unless you do work to ensure textures are correctly grouped by lifetime (which in many cases will depend on and diverge based on user input) you're going to end up with lots of 3MB holes.

You switch levels, and release many/most of your 3MB environment textures. But the icons remain loaded for your UI, so you don't free those. So now you have 3MB holes everywhere.

You start to load the next level. It's using a lot more alpha textures in it's environment - lots of water and glass. The alpha channels adds another 33% to the size - these new textures are mostly 4MB. None of these new textures fit into the 3MB holes. You've already potentially just wasted about half your available memory (or address space), with a single texture size in two formats of note. Within a single system ("render"). Oops.

Re:Memory and C++ Debugging at Electronic Arts [video] by danjayh on Thursday October 08, 2015 @11:18PM (#10357957)
OK. So the issue is the GPU then. That same scenario with data that could use virtual memory would be a non-issue, since you could mash the available fragmented phsyical ram together into whatever size blocks were needed using the MMU.
Re:Memory and C++ Debugging at Electronic Arts [video] by MaulingMonkey on Friday October 09, 2015 @08:33PM (#10363978)
> That same scenario with data that could use virtual memory would be a non-issue

Untrue I'm afraid - the case I'm basing this example off of could (and did) happen in virtual address space as well.

> since you could mash the available fragmented phsyical ram together into whatever size blocks were needed using the MMU.

Okay, so your physical memory is no longer fragmented. Unfortunately, your virtual address space still is. If I need to allocate a 3MB contiguous chunk of virtual address space (I do), I'm still SOL.

All the MMU gains you in this respect is a "free" memcpy - twiddle some MMU bits instead of copying data around from one buffer to another. It won't fix up any of your pointers.

Re:Memory and C++ Debugging at Electronic Arts [video] by danjayh on Thursday October 08, 2015 @01:48AM (#10351114)
Also of note is that fragmentation of physical RAM is completely irrelevant, aside from locality issues. They can map random physical pages into one nice, contiguous virtual space, and the only thing that matters is fragmentation in the virtual space. Since a each process gets its own entire 4GB virtual address space on a 32bit system, fragmentation nearly becomes a non-issue.

This is also why I suspect that they were mapping their virtual address space using block translation instead of a page table - if they had a page table, even in one process, it shouldn't have been a big issue since the virtual address space is many times larger than the available physical ram on the 360/PS3/Wii. The other possibility is that much of their RAM needs to be accessible by the GPU, for which fragmentation is an issue.

Re:Memory and C++ Debugging at Electronic Arts [video] by MaulingMonkey on Thursday October 08, 2015 @04:14AM (#10351452)
> Since a each process gets its own entire 4GB virtual address space

2GB of which is reserved for kernel shenanigans unless everyone enabled /3GB in their kernel boot parameters (they haven't.)

Default allocators start failing at around half that amount.

Meanwhile I'm running out of memory on the new generation of consoles where I have 5GB of physical memory and no virtual memory fragmentation to worry about.

Re:Memory and C++ Debugging at Electronic Arts [video] by danjayh on Thursday October 08, 2015 @11:20PM (#10357968)
Truth on the kernel. I'm used to working in an environment where we own the OS, the board design, and the design of many of the chips ... which means we can make it do whatever we need (downside is, you have to maintain it all too). However, our needs are unique enough to justify it -- computing is hard in an environment where atmospheric radiation can flip random bits for you (See: https://en.wikipedia.org/wiki/Single_event_upset).
Re:Memory and C++ Debugging at Electronic Arts [video] by roghummal on Thursday October 08, 2015 @04:14AM (#10351456)
I humbly suggest that if by some horrible twist of fate you had to work on games instead of "proper" software, you would quit programming.
Re:Memory and C++ Debugging at Electronic Arts [video] by nikanj on Thursday October 08, 2015 @01:36AM (#10351087)
By running everything in a single process and address space they can get n % more fps, where n>0.
Re:Memory and C++ Debugging at Electronic Arts [video] by danjayh on Thursday October 08, 2015 @01:51AM (#10351130)
Perhaps, but it's possible that it was actually the other way around since Virtual Memory and multicore came at about the same time. By running everything in a single process, they probably avoided forcing clean interfaces and dependencies between their functions, which might have actually hurt their ability to utilize the CPU efficiently.