would prevent me from taking on a spur-of-the-moment challenge outside my particular development area that might actually lead to some self-improvement
This sort of thing is why I got a chuckle out of the claim that there would be "enough interesting challenges in [OP's] job." No employer can offer me enough interesting work that I won't find something unrelated I'd like to play with in my spare time, and I'm probably not alone in that.
In my opinion, the work that most companies offer are challenges, but not very interesting ones and the no-outside-work clause would prevent me from taking on a spur-of-the-moment challenge outside my particular development area that might actually lead to some self-improvement.