My sense is that incels are red pilled and nu males aren't. So they might be similar in that they both fail to live up to a certain image of masculinity, but incels view that as a problem (or, at least, a hindrance) whereas nu males don't.

Red Pill comes from a famous late 90’s Sci Fi movie called The Matrix. The main characters world is not exactly what it appears to be at first and he is offered a choice of the red pill which would show him the truth or the blue pill that would keep him living the lie.

It has since been picked up by many communities as a way of saying they are ‘woke’ to the truth. It is currently in widespread use amongst the community of the subreddit The Donald by people claiming to have woken up from the liberal agenda or something along those lines.

In this context it probably refers to an infamous subreddit called the_red_pill that was a cess pool of pick up artists essentially training each other on how to hurt abuse and/or use women. It was banned sometime ago, during the Ellen Pao era I think. I believe they used it in the context of being woken up from the ‘lies’ of treating women as normal human beings if you want to find a lover.

Never hear of Nu Male, sounds like 4chan to me.

FWIW, r/TheRedPill definitely still exists.