At the time, it was a "Chinese problem". It's hard to "close your borders to China" without being accused of "xenophobia" in the modern era. What's your/their concrete counterproposal for the time? Leave them open? That certainly appears to have been the implicit counterproposal being made by the accusers; what else could they possibly mean? In hindsight, how has that fared?

Or perhaps close the borders with Brazil and other countries, just to avoid singling out China, even though at the time, the disease was only in China?

The accusation of xenophobia can only float in an environment where the accusers aren't actually responsible for making concrete policy proposals. Once you have to manifest the accusation in concrete proposals, then you're playing the "are you going to be ahead or behind?" game just like everybody else, and will lose in one direction or the other, just like everybody else.

(One of the privileges of being the minority party is the ability to take shots at the majority party without having to actually propose anything concrete that would allow counteranalysis. This seems to be a non-trivial component of how the parties tend to oscillate back & forth. I phrase it as the "minority party" precisely because it isn't an accusation I'm slinging at a specific party. I've seen the cycle several times now on both sides.)