I suggest you read on how modern display drivers are designed, you can't cross boundaries.

Designed. Many designs fail to achieve their goals. Has there been much research into 0-days in the GPU firmware itself? Or the closed blobs that get loaded into kernel space?

Yes, quite a bit actually, and you also need to understand that you aren't hitting the GPU directly by any means you are going through several layers of API's each with it's own security controls, then hitting a restricted end point in a usermode driver.

This is not to say that there aren't vulnerabilities in the drivers, that said the only 3 PE/CE vulnerabilities in the NVIDIA driver in the past 4+ years were not exploitable through any vectors you are suggesting since they involved NVAPI, AMD doesn't discloses vulnerabilities openly IIRC.

To exploit a vulnerability in the manner you suggest you need to break through the sandboxing and security model of the browser, break the sandboxing and security model of the web API you are using e.g. WebGL break through the security model and sandboxing of the actual API e.g. DirectX on Windows, break through the sandboxing and security model of the user mode driver, and then exploit a vulnerability in a kernel mode driver that might have actual access to something you might care about.

And even then it's not that simple in WDDM for example even in pure kernel mode you'll have issues accessing memory out of the bounds of your application due to how GPU resource are managed, pinned and translated.

To put it simply every process accesses a "virtual GPU" through it's own endpoint, there is a zero-out process which is invoked on both the GPU and system memory when any buffer is allocated or accessed, and there is an out-of-bounds behaviour control running on the GPU independent of the driver, basically once you access (read or write) out of bound memory the GPU would terminate the loaded kernel which would crash your application and the driver would be cycled (restarted).

The out of bound memory is a real annoyance anyone who's worked with GPGPU especially CUDA is pretty familiar with, it's the #1 app killer (as far as code errors go) and for good reasons, even a privileged kernel running natively on the GPU is protected from abusing it's own rights.

The CDM is sandboxed in Firefox and Chrome.

What does that mean in regards to parent's comment? If it renders, it went through both a graphics driver and GPU. Therefore they're in attack surface for malicious data designed to take over privileged code and/or DMA engines.

The CDM doesn't need to do any rendering, and interaction with the GPU can be limited to writing in a designated shared memory area.

Because it's isolated and there is no DMA really.

There is a reason why the graphics driver is in user mode, and why the memory is virtualized.

The application doesn't have traditional DMA unless you load a privileged kernel which you can't do via a browser.

An application can't access the memory of another application via the display driver this is a solved issue.