Though, from the wiki article:
Sir Michael Atiyah said of Witten, "Although he is definitely a physicist, his command of mathematics is rivaled by few mathematicians... Time and again he has surprised the mathematical community by a brilliant application of physical insight leading to new and deep mathematical theorems... he has made a profound impact on contemporary mathematics. In his hands physics is once again providing a rich source of inspiration and insight in mathematics."[6] One such example of his impact on pure mathematics is his framework for understanding the Jones polynomial using Chern–Simons theory. This had far reaching implications on low-dimensional topology and led to quantum invariants such as the Witten–Reshetikhin–Turaev invariants.
Even if his degree is in physics, and he's part of the "physics department" at IAS, I would classify him as a brilliant mathematician who decided to apply his energy to his passion, physics. And, come on, the guy won a Fields Medal.