I think the main difference is that eating an unfertilized egg doesn't end the life of a viable animal.

I am personally happily eat meat, but I respect people that choose not to kill animals and at least some of them find eating unfertilized eggs to be acceptable.