News for hackers. Stuff that matters.
Re:Fooling Around with Foveated Rendering by TeMPOraL on Tuesday October 06, 2020 @04:01AM (#24695608)
Tangentially related: I've recently been doing some VR work in Unreal Engine, and also reading Blindsight by Peter Watts, and this made me wonder: are saccades truly random? Could they be made predictable through some special way of crafting a headset, or through some stimulation? If so, then perhaps one day we'll be able to render only the pixels the mind can actually perceive at any given moment.
Re:Fooling Around with Foveated Rendering by codeulike on Tuesday October 06, 2020 @06:28AM (#24696315)
What makes you think saccades are random? When you are reading text, for example, they are methodical.

We have had machines that do what you describe for decades.

Here's Daniel Dennett in Consciousness Explained (1992)

When your eyes dart about in saccades, the muscular contractions that cause the eyeballs to rotate are ballistic actions: Your fixation points are unguided missiles whose trajectories at lift-off determine where and when they will hit ground zero at a new target ...

Amazingly, a computer equipped with an automatic eye-tracker can detect and analyse the lift-off in the first few milliseconds of a saccade, calculate where ground zero will be, and before the saccade is over, erase the word on the screen at ground zero and replace it with a different word of the same length. What do you see? Just the new word, and with no sense at all of anything having been changed. As you peruse the text on the screen, it seems to you for all the world as stable as if the words were carved in marble, but to another person reading the same text over your shoulder (and saccading to a different drummer) the screen is aquiver with changes.

The effect is overpowering. When I first encountered an eye-tracker experiment, and saw how oblivious subjects were (apparently) to the changes flickering on the screen, I asked if I could be a subject. I wanted to see for myself. I was seated at the apparatus, and my head was immobilized by having me bite on a "bite bar". This makes the job easier for the eye-tracker, which bounces an unoticeable beam of light off the lens of the subject's eye, and analyzes the return to detect any motion of the eye. While I waited for the experimenters to turn on the apparatus, I read the text on the screen. I waited, and waited, eager for the trails to begin. I got impatient. "Why dont you turn it on?" I asked. "It is on," they replied.

Re:Fooling Around with Foveated Rendering by TeMPOraL on Tuesday October 06, 2020 @07:37AM (#24696656)
> What makes you think saccades are random?

I didn't mean "random" as in governed by an RNG, just "random" as in we can't obviously predict their pattern. The opposite of that would be being able to tell when they'll happen and where they'll end before they start, and/or make them happen on demand and land on desired target.

> We have had machines that do what you describe for decades.

I didn't knew that. Thanks for citation. This leads me to ask: so why aren't we employing this for VR?

Re:Fooling Around with Foveated Rendering by codeulike on Tuesday October 06, 2020 @08:12AM (#24696855)
Ah I see
Re:Fooling Around with Foveated Rendering by ramy_d on Tuesday October 06, 2020 @09:22AM (#24697385)
wow, is there a video of this? sound really interesting
Re:Fooling Around with Foveated Rendering by modeless on Tuesday October 06, 2020 @11:59AM (#24699137)
Saccades are neither random nor very predictable. They are voluntary. You can predict the "salient" parts of an image that humans will prefer to look at, and use that information to e.g. improve video compression. But your predictions will always be wrong sometimes because humans are unpredictable.

It should be feasible to detect saccades instead of predicting them, and render new pixels more quickly than the eye can move. But it is right at the edge of what's possible, and really needs more reliable eye tracking than is generally available today, along with purpose built rendering techniques and higher resolution + higher field of view displays.

Re:Fooling Around with Foveated Rendering by RobotCaleb on Tuesday October 06, 2020 @11:30AM (#24698761)
One of my favorite books. You might be interested in a fan movie launching next week. http://blindsight.space/

https://mailchi.mp/b4c8b26e025d/blindsight-project-update-se...

Re:Fooling Around with Foveated Rendering by TeMPOraL on Tuesday October 06, 2020 @12:57PM (#24699946)
Thanks, very timely! I'm in the middle of reading the sequel/sidequel, Echopraxia, right now.
Re:Fooling Around with Foveated Rendering by 082349872349872 on Tuesday October 06, 2020 @05:26AM (#24696029)
Not prediction, but: there used to be an Exploratorium exhibit with a large screen which updated randomly. One could toggle between flashing (erase to black) updates and smooth (just overwritten) updates.

The difference in sensation of change was palpable.

Re:Fooling Around with Foveated Rendering by kebman on Tuesday October 06, 2020 @07:36AM (#24696645)
Are you talking about Critical Flicker Fusion here?
Re:Fooling Around with Foveated Rendering by 082349872349872 on Tuesday October 06, 2020 @07:48AM (#24696719)
Possibly. (I'm unfamiliar with the term, TIL!) As I dimly recall from last century, we are sensitive to flicker in the coarse-resolution field, but insensitive to non-flickering change outside the fine-resolution field.
Re:Fooling Around with Foveated Rendering by kebman on Tuesday October 06, 2020 @09:20AM (#24697368)
Then you're probably familiar with the 24 frames per second cinema standard for projecting films. Except each frame is actually projected three times, leading to a shutter speed of 72 frames per second. (In reality there are far more options today with IMAX and video tech, though.)

Even though there's only 24 images on the film per second, this triple exposure results in a smoother experience for the viewer. This is due to how critical flicker fusion works (a.k.a. persistence of vision). [1] (Though another reason for doing it is simply so that the film won't be burnt, since those xenon gas projection lamps run very hot.) See also beta movement and phi phenomenon. [2]

These physiological phenomena are of course very important to consider when making VR devices and games.

In order to create the effect mechanically in the film projector, each frame has to be stationary before the shutter is opened. If not, all you'd see is a blur. Thus, when the film is pulled forward, the shutter blocks the light from projecting the image onto the screen. In fact, during some half of the movie, people are actually sitting is pitch black darkness. Think about that next time you go to the movies! ;)

Please note that peripheral vision has a higher sensitivity to flicker than foveal vision. I'm unsure how interlacing affects that, though, if applickable. This effect might also be different on video systems where various forms of interlacing may or may not be used.

[1]: https://en.wikipedia.org/wiki/Flicker_fusion_threshold

[2]: https://en.wikipedia.org/wiki/Beta_movement

(Please excuse my lack of sourcing. Most of this is off the back of my head, and from books I am no longer in posission of... But the Wikipedia links should give you a good start.)

Re:Fooling Around with Foveated Rendering by TeMPOraL on Tuesday October 06, 2020 @10:33AM (#24698049)
> Thus, when the film is pulled forward, the shutter blocks the light from projecting the image onto the screen.

Ironically, it sounds like a complement to saccades! Where saccades involve shutting off vision when your eye moves, this involves shutting off projection when the image moves.

Re:Fooling Around with Foveated Rendering by lopmotr on Tuesday October 06, 2020 @06:05AM (#24696225)
Would eye tracking be fast enough to follow saccades and generate the high resolution parts on the fly? I assumed it would and we'll eventually have VR at resolution of human perception done that way. I hope so!
Re:Fooling Around with Foveated Rendering by mncharity on Tuesday October 06, 2020 @07:29AM (#24696611)
> Would eye tracking be fast enough to follow saccades

MEMS kHz eye tracking enables even predicting where a saccade will land 20+ ms before: https://www.youtube.com/watch?v=JEg4l5KuQgI&t=452 (AdHawk @ AWE 2018).

Re:Fooling Around with Foveated Rendering by cjsawyer on Tuesday October 06, 2020 @12:50PM (#24699860)
All of the current products are basically cameras, right? I’m excited to hear about possible improvements
Re:Fooling Around with Foveated Rendering by mncharity on Tuesday October 06, 2020 @06:49PM (#24703295)
All that I know of at least?
Re:Fooling Around with Foveated Rendering by TeMPOraL on Tuesday October 06, 2020 @06:09AM (#24696235)
Eventually, maybe? But I was hoping there would be a way to cheat and somehow force saccades to happen on a pattern that can be predicted beforehand, so that we could render only the parts of the image that the eyes are just about to jump to.
Re:Fooling Around with Foveated Rendering by lopmotr on Tuesday October 06, 2020 @08:16PM (#24703997)
Probably to some extent. Movie makers anticipate what part of the screen viewers will look at. They can control it with content, motion, focus, etc. Maybe you're hoping for something more automatic? For a simple looping video we might get bored and our eyes start wandering around, but if we're paying attention to the content, we probably keep focused on the moving/important objects.
Re:Fooling Around with Foveated Rendering by TeMPOraL on Wednesday October 07, 2020 @02:22AM (#24705768)
Yes, I'm hoping for something automatic enough to allow for a turbocharged version of Foveated Rendering. One where, instead being dead ahead of you, the region of high-resolution rendering would follow your eyes as they saccade - so you'd never be able to tell the headset renders only a tiny fraction of the image at any given time, while giving it even more time to produce a high-quality render there.