Wavelength specificity. You only get the original focal depth at the wavelength you used to make the hologram, and it changes proportional to the wavelength. When you consider that blue light is 2/3 the wavelength of red light, you can see that this is chromatic aberrations big brother.

Diffractive elements are very wavelength dependent so while something like this would only work for monochromatic light (like a laser), but not for multiwavelength light. For that you would need to do some colordiversity.