Depends on the relationship and contracts in place. If he said employer#1 and employer#2, yes, that's a problem.

"Customer" implies a much looser relationship, but you should still make sure appropriate contracts defining ownership are in place.