With a big emphasis in "small"... for a raytracer you don't need much more than a 3D vector with addition, multiplication, and dot product. For reference, the "linear algebra library" in my tiny raytracer [1] is 6 lines of code (and two are method closing braces): http://gabrielgambetta.com/tiny_raytracer_full.js (functions dot and A_minus_Bk)

[1] http://gabrielgambetta.com/tiny_raytracer.html