While batching (and reordering) can have a benefit overall for rendering, it can have a significant (depending on how thoroughly you do it) overhead, and Disney are primarily doing it because they use Ptex to do texturing, instead of the more conventional and more widely used UDIM texture atlasing method for assigning textures to meshes.

Ptex really chokes with non-coherant texture accesses (partly due to the anisotropic filtering method it uses) in terms of thread scalability, so Disney have gone to great lengths to get around this issue. Doing such accurate texture filtering (the same goes for standard UV EWA filtering) is technically better, but is expensive, especially in a path-tracing context, where the whole point is to amortise the cost of shading over all the rays by making each intersection / shading calculation as cheap as possible.

From what I hear there are big downsides to this current implementation: the streaming of batches is pretty much off-line, so time till "first pixel" is significant, and thus you don't get interactive rendering functionality.

Awesome information, thank you very much!

The batching that Disney is doing is a very recent direction and not something that is common - PBRT serves as a straight forward reference implementation.

You are right though that it would be fascinating to see the speed difference. My guess is that it would be substantial.

Even compiling PBRT with Intel C++ instead of Microsoft's C++ compiler gives a %30 - %50 speedup, likely due to reordering instructions for better memory coherency on a granular level (batching rays would give better memory coherency on a very broad level).

It would be interesting to see PBRT re-written with C++ 11/14 idioms, and look at how that affects the execution speed as well.

That's also true, PBRT uses a lot of virtual functions.