I finally realized it after wondering why I looked 'worse' in photos with friends, when the rear camera is being used to take the picture. That's when I did a rear v front photo comparison.

The face smoothing effect is subtle, but it's definitely there. It's not as extreme as "filters" that can basically change your skin tone and texture.

And it's possible that Samsung does this and not others. My last Samsung phone had this as well, and I usually buy from them.