Because we have extensive samples of the Neanderthal genome extracted from bone fragments, and before that from morphological studies of the Neanderthal skeleton which display far greater differences from humans than any differences in skeletons between human races. Even 18th Century observers were able to see that.

Yes there is no one universally agreed definition of species, but Neanderthals were vastly more different from living humans than and differences between living human populations.

BTW is there a good comparison for how big the anatomical differences are between Neanderthals and modern Europeans, compared to the differences between modern Europeans and modern, significantly distinct populations such as Congo pygmies?

https://en.wikipedia.org/wiki/Congo_Pygmies

(Anatomic differences are perhaps not as important as cognitive and psychological differences, but the latter are very hard to find out about extinct populations.)