In fact the foundations of IQ tests lies heavily on the eugenics movement which wanted to prove the superiority of able bodied white men. They actually started by assuming brain size correlates with intelligence (it doesn’t) and then set out and proved that on average white men had the biggest brain of them all (which is also false). The nuances of fitness does not help in this goals as one can easily just pick a type of intelligence/fitness which white men perform poorly at in general so their claim was that there exists a general intelligence to remove this nuance. Their current IQ tests are all set up to assume this is true, and hence you can see it in the results. If you can’t then you are not really measuring intelligence.
Books such as The Bell Curve are not in and of them self racist. This book in particular makes zero racist claims. However it relies on deeply racist studies as its premise, so if you want to conclude anything from it, that conclusion is racist.
I don’t think every proponent of IQ is racist, however there is a lot of racism embedded in the research, sometimes directly, but most often by association.
Thing is, building the ontology is the most important part of the scientific process. Otherwise you're just praying to a blind idiot god and listen into the void for answers. Or as the author puts it, "statistical arcana."
To be concrete, I'm pretty confident I overtook half of humanity in that fitness test in the last year. Not because I was bitten by a radioactive spider, but because I started to work out. Now to claim the analogy to intelligence, that would need an answer to what the analogy of working out is in intelligence.
Or to look at it on the level of people who compete in a sport, I'm quite confident that Tom Brady is not very fit by NFL player standards. Similar for Xavi in soccer, but Christiano Ronaldo is probably near the top, and I don't have a good intuition in case of Messi, because he is a quite small player and that will have an effect.
The preceding criticism is enabled, by having a good intuition of what the fitness test is, I have a pretty good intuition about the mechanics of a push up. That is not there for intelligence, and I strongly suspect that this whishi-washiness is whats actually holding up all arguments about iq, since it prevents concrete criticism.