They used JPEG XR: http://en.wikipedia.org/wiki/JPEG_XR (Microsoft proprietary algorithm).

You can't really compare it to skyrim though. Skyrim doesn't use mega texturing (or virtual texturing). It repeats textures all over the place. In rage, there is a huge texture for everything. IIRC, the uncompressed textures performed a total of 1.5TB or something like that, so I would say that they did an awesome job with the compression.