I believe part of my issue might be that I'm using 1 HDMI port and another mDP to HDMI converter. From what I've read it seems like people tend to have these problems when using converters. So I guess if I were running directly off of mDP I'd probably be having a better time.
The point being that anecdotally, windows doesn't have it down either.