You'll actually find it pretty easy to learn both. Elixir was very inspired by Clojure, so it works similar to it in a lot of ways. It is like taking Clojure and removing the Lispy aspect from it (which to me is the biggest downside of Elixir, as I love Lispyness, but that also makes it more approachable and familiar to people without a Lisp background).

The biggest difference will actually lie on the host platforms. Learning BEAM and Erlang standard library Vs learning JVM and Java standard library.

For complenetess, I want to point out there is also a Clojure variant running on BEAM/Erlang called Clojerl found here: https://github.com/clojerl/clojerl

Development is pretty active, and I think it is approaching maturity pretty soon.