I think the main difference is that eating an unfertilized egg doesn't end the life of a viable animal.

I am personally happily eat meat, but I respect people that choose not to kill animals and at least some of them find eating unfertilized eggs to be acceptable.

In my personal worldview I view unfertilized eggs as approximately morally equivalent to eating a peanut. In practice it's difficult to make them exactly equivalent but by carefully selecting where the eggs come from you can get closer.