I had considered making a ray tracer that worked that way, but with the slight difference that rays didn't have to hit the camera, but would just have to hit a point within line of sight of the camera. Obviously there would be massive gaps between pixels, but I would fill it in with a Voronoi diagram [0], or perhaps shaded with Delaunay Triangulation [1]. This renderer would be nothing more than a toy or proof-of-concept, and not intended for real usage.

The classic FOSS ray tracer, POV-Ray, can actually do this. You can define a light and define an object, and it will shoot rays from the light to the object and trace each ray through refraction and reflection. With this, you can simulate the way ripples in a pool concentrate light on the floor of the pool [2] or bending and refracting [3], without manually calculating it and adding extra light sources.

[0] https://en.wikipedia.org/wiki/Voronoi_diagram

[1] https://en.wikipedia.org/wiki/Delaunay_triangulation

[2] http://www.antoniosiber.org/bruno_pauns_caustic_en.html

[3] http://www.povray.org/documentation/view/3.6.2/424/

Yeah, there are multiple techniques that do this sort of thing. Photon Mapping and bidirectional path tracing are on ends of the spectrum

There are two-pass approaches that do this, such as photon mapping. In the first stage, light is emitted from light sources and allowed to to scatter throughout the scene, and each "hit" of a photon on a medium is stored in a large data structure. Then in the second pass rays are traced out from the camera, and on each medium the ray intersects, nearby "hits" are used to estimate the light coming from that spot. This produces effects like caustics efficiently.

It's not really ray tracing, but some engines do something analogous to render shadows. Rendering the scene from the point of view of the main light source, and just recording the depth value at each pixel, yields a shadow map. Now for the main render, for each pixel rendered, re-project its coordinates into the light source's camera and compare the distance to the shadow map. If it's further, it's in shadow.

Sure, you could compute shadows with forward ray-tracing, but it's still generally more efficient to compute shadows with rays originating from the camera end of the path.

Of course, this all starts to get a little more complicated when trying to compute global illumination.