It's been a long time since I've read it, but I vaguely remember a write-up many years ago (before we used ML for NSFW detection) about how either yellow images would often produce false positives in popular detection algos, and/or about how the yellow-ness of an image could be used as a lower-tech proxy for basic NSFW detection (seemingly because of its proximity to some common skin tones).

I don't know if either of these are relevant here, but it's possible a weight on Yellow Images could be related to NSFW images.