Usually one starts at the eye and traces back through the scene. But the converse is also sometimes useful, and the terminology is slightly ambiguous (different people have called different variants "forward" and "backward" at different points in time...), Wikipedia has some details:

http://en.wikipedia.org/wiki/Ray_tracing#Reversed_direction_...

> Am I completely wrong here?

No.

> Is my whole world a lie?

Yes.