By the same logic, African-American never comes up without race being a significant part of the context.

I'm mostly surprised because "African-American" is a mouthful compared to just "blacks", so I'm suspicious that in common language it is somehow limited only to racists.

"Blacks" is also more accurate if you just want to refer to people by skin color (for whatever damn-fool reason).