Nginx Lua bindings does have direct access to OpenSSL primitives (or LibreSSL, or BoringSSL). I have also created (LuaJIT) binding to libnettle [1] that you can find from here [2]. (libhogweed, aka rsa/dsa bindings, and documentation are still a work in progress).

[1] http://www.lysator.liu.se/~nisse/nettle/ [2] https://github.com/bungle/lua-resty-nettle