I made a quick video explaining how the webapp works: http://youtu.be/Z2ZXW2HBLPM

Essentially, using the gridPositions.json to populate a Voronoi Grid (toggle display of the <canvas> in your browser). Then your cursor position is used against the grid to find which shape encloses it which maps back to the source JPEG which is then displayed.

I watched your video. It does not seem to explain how the fingertip position in each photo was calculated, which it seems to me is really the heart of the magic.

Yes, I was wondering the same thing. The difficult engineering problem here is first assigning pointing locations for every photo in the database. Because he didn't explain this step, I guess he just manually assigned each location for every photo.

That's it, the developer has defined the pointing position for every photo manually.

That's what I assumed until I pointed to ~20x20px from the top right corner and got a picture of a house in a flood.

He says how it's basically a json file with every image mapped to coords of each voronoi "tile". I guess it was handmade in advance

Not a bad guess but I'd be interested to actually know the details. Making it all by hand would be problematic for reasons beyond the sheer volume. How would one find a deep, diverse set of images of people pointing? How did the creators ensure they had images to cover all possible pixel positions within a certain proximity? It seems like it would take reviewing many more than 900 images to produce a final set of 900 that includes even coverage.

Another guess here, but it could have been crowdsourced using Amazon Mechanical Turk. Assuming a conservative 0.05$ by picture, the total cost would be 901 * 0.05 = 45.05$.

Yeah, this seems likely. Here's how I'd do it: create a webapp similar to this one to show to your Turkers. The difference would be that it would show an image at random, and the Turker would decide whether or not someone is pointing in the image, and if so they would click where the person was pointing. For each image, take the mean position of mouseclicks. This becomes a seed for the Voronoi diagram.

You can tell he/she "cheated" a bit for some the parts of the canvas that didn't get many seeds. For example, if you move your mouse within one of the large cells in the bottom left, the image just moves to keep up with the pointer. :P

But then everyone in the photos would be Indian.

> How did the creators ensure they had images

You don't need a different picture for every single pixel: if the picture is large enough, you can easily crop it and use it to cover few thousands pixels.

They could have had help from friends. Also, the design of the application makes it very easy to add new images to the set (just throw a new entry into the JSON file containing the x and y coordinates of the finger, and the script does the rest whenever the user loads the page).

But then, why do I get a picture of some garden when I move my pointer in the top right corner?

You can just barely see the tip of a finger coming in from the right in that one.

If anyone doesn't care to explore the DOM themselves, this will show the Voronoi diagram and make the photo on top of it transparent. For Chrome, Ctrl+Shift+J and paste it into the console.

  document.getElementById('data').setAttribute('style','');  
  document.getElementById('content').setAttribute('style','opacity:0.7;left:0;top:0;margin:0');
  document.getElementById('underlay').setAttribute('style','left:0;top:0;margin:0');

Interesting example of how a hard problem (image recognition/understanding) can be faked for some cases by solving an easier problem (lookup table, a few hours of manual intervention.)

Seeing http://www.pointerpointer.com/gridPositions.json really took away some of the magic for me. Still, very clever and shows that web magic can be in the eye of the mouse-holder!

Erm.. even there were some computer vision algorithm involved, he would probably want to run it offline.

So there's no advanced computer vision algorithms involved :(

doesn't really work for me with Chrome on Windows 7 - an image flashes for a split-second, then it's back to "Finding pointer" (I'm not even touching my mouse)

Same here. Just flashes the picture really fast in the latest stable release of Chrome under Windows 7. Works beautifully in the latest Firefox though. Windows IE isn't supported at all apparently (I tried IE 9).

Are the developers willing to share some insight into how this works? The picture selection is right on and there seems to be a good number of pictures at that.

Agreed - never quite settles to actually show anything, just keeps asking me to hold still even though my hand's off the mouse...

Same problem windows 7 x64 Chrome 19.0.1084.52

It seems to work for me when I move the cursor off the page, though that misses the point.

Works fine in Chromium 21.0.1136.0 (136796).

Snow Leopard/Chrome 19 works fine.

Seems like it's Win/Chrome.

intersting: on Ubuntu 12.04, still with Chrome, it works

doesn't work on Windows 7 Chrome 20.0.1132.27

It looks like Chrome is firing the "mousemove" event when it shouldn't.

One way to make it work is to right click where you want it inside the square it and then left click anywhere outside the square.

In the Javascript looks like the error could be patched in Flasher.js line 72; save the last event.clientX and event.clientY and then check if the mouse coordinates have actually changed.

Doesn't work in Chrome but the 'right mouse click' hack works.

There are likely some good opportunities for this code however speed of response may be a problem? Runs a little slow on my PC.

Fun :). I did something similar with a bunch of photos of a stuffed monkey. I made a music video instead of an interactive web app, though the latter would probably not be too difficult since I wrote the software using Processing. http://sporksmith.wordpress.com/travels-of-code-monkey/

A faster (older) version of this by the same people, with camera flashes: http://flitser.org/

Warning: at least one of the images is not "family-friendly". Are the images automatically collected?

It's probably easier to automate the picture harvesting here via an intensity filter. I'm still stumped about how the finger-recognition was accomplished in the OP.

Definitely good for a chuckle. It helps a lot that the photos are very well aimed, really makes it feel magical :)

The images are adjusted. I was able to get the same picture 3 times in a row by moving cursor slightly, but the finger was always aligned exactly the same on the cursor, even though I had moved it.

Just wasted about 20 minutes of my life playing with this. I would really like to know the technology behind. Is there some sort of ML going on, or is this human trained? Either way, very entertaining and scary accurate.

Something as low tech as having a predefined map of {location : picture} would do it. Break up the screen into a grid of NxN quadrants, like 20x20. Then just find 400 pictures with a finger near each quadrant. Translate cursor to a quadrant x,y; look up picture. Done.

You wasted 20 minutes ... what makes you think someone didn't waste 20 days pulling the image collection together. Occam's razor ;)

All the images are listed here:

http://www.pointerpointer.com/gridPositions.json

There are around 900 of them! I'd have used Amazon's Mechanical Turk...

Very cool effect, I also wasted way to much time playing with it :)

Good "point" -- the right question posted to mturk/crowdflower/etc. could have knocked out that json file for relatively minimum time/cost. Oh, the things we humans engage in.

That's what I assumed. If I just moved my mouse a couple pixels I usually got the same pointing image back. I think they just divided it up into regions and then load the appropriate image for each region.

However, if that's all they're doing, it seems odd that it takes so long to load... Maybe some of that's just a built-in delay to keep from constantly cycling images from someone who's bumping the mouse.

The delay might be to build up suspense. The two–three seconds it spent loading I spent thinking "how could something that's essentially xeyes possibly take this long… ohhh haha very cute"

There are serious problems around the upper right corner. I did not get any results from there. I'm wondering whether the pictures are originally pointing that coordinate or are they resized+translated to some offset?

You'll have to move the cursor left a bit, to get the same image, but with visible finger. The images are indeed offset so that in the upper right corner the finger actually doesn't fit in the frame.

I don't see the point of this. The images load far too slowly.

One suggestion is to move the jquery from jquery.com to your domain for two reasons:

1: It will be one click less, for those who use noscript. 2: It will allow your site to survive the case that jquery.com changes anything, or goes down.

Besides that its a nice toy.

Alternately, use the Google CDN.

What is the point about this beyond showing pictures where the finger is next to my cursor? (Why is this popular?) It's not more interesting than a circumference search. You could implement that with a simple rtree.

Haha, very clever. Although I seem to be running into problems whenever I put the mouse into one of the 4 corners of the screen. It can find anything more centrally located, though (apparently?)

I was actually expecting that somebody did some machine learning / matching over a large set of pictures and found all those images / coordinates.

This is pretty neat, I wonder if it has to be human trained or if there's some sort of computer vision algorithm determining pointing.

After looking into to the code I found out that this app is made by using Voronoi diagrams. [1]

The actual positions are saved in a json file. [2]

[1] http://en.wikipedia.org/wiki/Voronoi_diagram

[2]http://www.pointerpointer.com/gridPositions.json

Yes, but how did the author create that .json file? Manually or using some ML algorithm?

What would be interesting is if new images were periodically added and the grid adjusted in response.

How can I get a pointer to my cursor in Javascript? I might find this useful in automating certain processes.

On Firefox, never goes past "Pointer found, pointing". No image loads, works on Chrome though.

What's this? I saw some pictures flashed after some loading indicators are shown...

This is technically interesting, but voyeur-istically creepy.

Doesn't work in IE9. That was the limit of my experience :(

Is this a some kind of concentration game?

Why does it take so long to load?

I was wondering the same thing. With a precomputed voronoi diagram and only a few hundred cells, lookups should be almost instantaneous.

Doesn't load for me, on Chrome/FF/IE9...

awesome!