Does that help them? It makes the trade that at sports the abuses less financially viable, but that might make the abuses worse. If the trade stops they lose their jobs and money stops flowing from wealthier regions to poorer regions.

Fair trade is better than no trade (other things being equal) if you want to support foreign workers.

Just global trade systems are a hard problem, as I see it, and really can't be organised from the consumer end (ie by simply choosing where/if to buy goods).