I think thats what actually happened. We only have DNA from either male or female Neanderthals, forgot which one though. As far as I know if the neanderthals were a certain gender then the hybrid would be infertile.