Definitely go for it, then, and try to do Common Lisp rather than anything else. You can always expand later to JVM-based Clojure, or indeed contract to Scheme, but it's good to start with the thing itself.

The world always needs more good Lisp programmers.