You could pack the raytracer and the scene into a GLSL fragment shader without any trouble whatsoever and achieve something greater than 30fps out of the gate.

Check out https://www.shadertoy.com/ - all of this crazy stuff is generated using two triangles and a fancy shader. Almost every 3D scene there is generated using some form of ray casting or ray tracing on the GPU.

It does that by calling in to numpy which is native.