My experience is similar, though nngroup.com has some reports that look like an okay research (though unreplicated as well). NASA's color graphics pages [1] seem to be based at least partially on research, contain further references, and generally seem good to me, but they are focused primarily on accessibility (and similar to WCAG). Searching for relevant papers on CiteSeerX, Google Scholar, and so on usually yields surprisingly little.

[1] https://colorusage.arc.nasa.gov/issues.php