It's not an empirical science, but it is a science, where "science" means any systematic body of knowledge of an aspect of a thing and its causes under a certain method. (In that sense, most of what are considered scientific fields are families of sciences.) Mathematics is what you'd call a formal science with formal structure and quantity as its object of study and deductive inference and analysis as its primary methods (the cause of greatest interest is the formal cause).