Wouldn't other semiconductor materials allow higher clock speeds and it's just that the switch would be difficult to do incrementally?
Short answer is "no", but other materials may allow smaller manufacturing technologies. "Incrementally" isn't really a factor - every die-shrink is basically a new manufacturing process.
http://arstechnica.com/gadgets/2015/02/intel-forges-ahead-to...
http://arstechnica.com/gadgets/2015/07/ibm-unveils-industrys...
Wouldn't other semiconductor materials allow higher clock speeds and it's just that the switch would be difficult to do incrementally?