It seems that to handle this you should have lines between the label and the slice instead of a legend. Google Sheets seems to do this by default but it seems to sort colours from dark to light which may make the boundaries of some colours hard to see for colourblind people. I would assume that putting the slices light-dark-light... would be optimal for the contrast between them, even for people who are not colour vision deficient.