Full light field displays have to draw every ray, not just every location; that's two additional dimensions. Resolutions scale badly, and if you make any of the four dimensions lower resolution, you'll get ugly artifacts.
Microlenses aren't the limitation. Pixel density is, both in physical manufacturing side and when refresh / drawing.
I keep thinking there will be a significant reduction in complexity there. The intensity of a pixel in a hologram is essentially an integral over all surfaces visible from that point. So imagine a rather complex formula applied for each surface for each pixel. Then imagine holographic bounding boxes that compress complex geometry into a few holograms of what's inside. This would reduce the n^4 back down, but the resolution required for holograms is still very very high. But we could use fancy GPUs to evaluate the integrals.
Just hand-waving thinking here...
But compressing a light field and projecting one into an eye are totally different things. Your display needs to be capable of displaying all 3n^4 possible intensities at different times. Depending on the scene and where you're looking, you can get away with showing only a smaller handful of those (some megapixels) but the display still needs to be capable of displaying them all.
If your display is just fixed LEDs behind a microlens array then you still need n^4 resolution. Like, a megapixel per pixel. Most of the pixels can be off at any given time, but you'll need them all to display arbitrary scenes.
Otherwise, the existence of those additional rays is what allows for focus accommodation: as you focus in different planes, it 'shuffles' the light around, to create sharp edges (where similar light rays line up) or blurry foreground/background (similar light rays strike different portions of the retina -- and if any are missing, there is a 'hole' in the blur disk).
Pedantic, perhaps... but these are the stakes.
I think Google is on the right path with Tango-first. Didn't used to think so.
Items in infinity emit parallel rays, items very close to the eye emit rays with a large spread...
Those four dimensions are projected into two on the retina, but the exact projection is a function of where the person is focusing. (That's how we understand focus; by changing the lens parameters, we can focus at different depths in the scene.)
So three dimensions: two for position and one for "amount of off-axisness".
Any data compression method has to be compatible with the underlying display technology so you can bring the signal straight into the modulator. In addition, there's a wire problem: you can only bring so much signal into a light modulator because the signals interface through a surface, not a volume.
For these reasons and others, true holographic displays are simply not competitive at this time with stereoscope-like VR systems, which rely on tried-and-true, market-driven incoherent-light raster display technology.
People have been asking these kinds of "why not" questions about 3D and building various types of 3D display technology in garages and labs for almost two hundred years. With the exception of the discovery of optical holography, it's surprising how much is fundamentally unchanged.