It certainly seems plausible, but I wouldn't entirely rule out other possibilities.

Do to give an example if you present the LLM with two people that are exactly the same except they have different color shirts I think it will suggest slightly different salary for one than the other for no clear reason and without any obvious bias in the training set.