It is also used in WebKit's JSC. Using the NaN space goes back much further than LuaJIT. E.g., you can see it mention in this 1993 survey paper: http://lambda-the-ultimate.org/node/3912.