I would associate that more with "employee," with "worker" being an umbrella term. Note that "worker" is also used in a bunch of other contexts. Worker bee, worker thread, miracle worker, etc.

Okay but using umbrella terms obscures the crucial aspect of being a slave: non voluntary, unpaid labor. Yes, it is technically correct in an extremely literal sense. But that's not the same as it being the right term in the context of teaching students about the slave trade. If I said "his great grandfather was a worker", it would not imply slavery, indeed most people would assume the opposite connotation, that his great grandfather was a paid worker.

I can't help but feel like you're confusing literal correctness with appropriateness. And while literal correctness is an easier judgement to make, there are many literally correct statements that absolutely miss the point. Saying that slaves were "workers" is one of them.