The "Schwarzer Mann" of the children's game doesn't have anything to do with race. Historically it was a mythical figure used to scare children (Kinderschreck) comparable to the bogey man, visualised as a man with black clothes or a shadow creature not a man with black skin color.

There is a theory which connects him to the black death also, which would fit to the game mechanics.

https://de.wikipedia.org/wiki/Wer_hat_Angst_vorm_Schwarzen_M...