News for hackers. Stuff that matters.
Re:A Face Recognition Algorithm That Outperforms Humans? by gcr on Wednesday April 23, 2014 @10:15PM (#7638163)
I'm a computer vision grad student. A few things concern me about this work. Maybe they're incidental, but I'm not ready to throw my hands up in the air quite yet.

- Why wasn't this accepted to CVPR/ECCV/one of the well-established computer vision conferences? I would love to read some of the reviewers' comments about this work before I give further judgment. (If this really is some CVPR preprint, or if it actually is peer-reviewed, I'd feel much better about this.)

- Why isn't this work listed on the official curated "LFW Results" page that Erik Learned-Miller maintains? http://vis-www.cs.umass.edu/lfw/results.html Is this work so new that Erik hasn't had time to review it yet?

- Human performance on LFW is 99.2%, which is higher than what the authors think it is. The performance drops to the (claimed) 97% when we only show humans a tight crop of the face: http://www1.cs.columbia.edu/CAVE/publications/pdfs/Kumar_ICC... They discuss this difference in a paragraph in their conclusion, but I consider it dishonest to use the lower number in the abstract and imply it in the title. In fact, I consider it misleading to put "Surpassing human performance" in the title to begin with, but that's another matter :)

- Showing good performance on one dataset (LFW) is certainly not enough to show that this "outperforms humans" in the general case. Getting a state-of-the-art result on LFW these days is like squeezing a drop of water out of a rock; in my opinion, we should turn our attention to harder datasets like GBU now that these "easier" ones are solved.

I'm not terribly familiar with Gaussian processes so I'm not sure whether the math works out, but it is a pretty uncommon thing to try in this domain. (Perhaps that's what makes this work interesting, especially since this year seems to be the "Deep Learning is Eating Everyone's Lunch" year)

I also wish they describe what final-stage classifier they use for the "GaussianFace as Feature Extractor" model. Often, that's the most important step; it's strange that they didn't compare with POOF/High-dimensional-LBP/Face++'s deep-learned features/any of the other state-of-the-art feature extractors, especially considering how much worse "GaussianFace as a binary classifier" does (93% vs 97% is a huge difference in this dataset)

Just my two cents. It definitely demands further exploration. I don't see any obvious mistakes, but I'm not sure why their approach works as well as they claim it does either.

Edit: I don't mean to start a witch hunt or anything, but if the authors have the guts to put "Human-level performance" in their title, they're just begging for the community to inspect every detail and point out all the flaws in every minutiae in their work. It's our community's hot button. It's similar to the old adage about how if you want a Linux user to help you, you have to tell them how much Linux sucks. That's where much of my skepticism comes from. The most astounding papers are often the most humble, but "humble" certainly doesn't describe this work.

Re:A Face Recognition Algorithm That Outperforms Humans? by apu on Wednesday April 23, 2014 @10:51PM (#7638269)
In some sense, the 97% is indeed the more fair number to compare against, assuming that this paper also restricts the algorithms to only see tight crops of the face.

Obviously using more than just tight crops will give you more information, but our point in differentiating those cases when measuring human performance was that the way LFW was constructed gives you MUCH more information when using loose crops than "normal" images would. For example, many images of actors are at award shows (the Oscars in particular), and so if you see that kind of background in a pair of images, you can just say "same" and have a very good chance of getting it right. That's what the "inverse crop" experiment shows [1] -- when you block out the face in LFW images, you can still get 94% accuracy!

In normal images, however, the background won't normally give you so much information.

I do feel somewhat bad that our human verification performance experiment numbers are now being used to create linkbait titles like, "computer algorithms can beat humans," because that's obviously not true (nor have I ever believed it), but in my defense, in 2009 we didn't really think about what the press would do in 2014 when algorithms started saturating on LFW =)

[1] http://homes.cs.washington.edu/~neeraj/projects/faceverifica...

Re:A Face Recognition Algorithm That Outperforms Humans? by egmil on Thursday April 24, 2014 @10:36AM (#7640493)
Hi,

gcr makes a number of excellent points here. I completely agree about the human performance comments. Human performance on LFW is 99.2%. While some people claim this is "not fair" because it includes some background of the picture, this is the way the benchmark has been defined. There is no universally accepted way to define the face verification problem, and when we built LFW we chose to define the task on these images (without cropping them closely). Not surprisingly human performance goes down when the images are cropped more closely to the face. My response to people who say that it is unfair to use the whole image is "go ahead and write a program which takes advantage of the whole image!"

I have not read the paper carefully yet, so I cannot comment on the technical content. However, people have raised the question of why it's not yet on the LFW results page. First of all, we don't put up results until we received the results file from the authors yet, and we haven't received such a file yet, so of course the results are not up yet. Second, even if we receive results files from an author, we require that the paper either be published in a peer-reviewed conference or journal, or, alternatively, that if it is commercial, that an executable be available so that the system can be tried by other experimenters. If we are sent results from a paper that has been accepted but not yet published we sometimes show the results on our results page, but highlight it in red to show that the paper is forthcoming.

Additionally, Gary Huang, who does a fantastic job maintaining the LFW site, is redoing the entire results page to better present the information, and so results may experience a short delay while we are working on the new page.

One interesting thing to note about methods that are getting very high accuracy rates on LFW is that they typically use enormous amounts of outside training data (like the recent DeepFace method by FaceBook researchers). Part of the purpose of our reorganization of the results page is to highlight better the differences between methods that use outside data and those that use only the LFW training data.

The community should remember that a K-nearest neighbor classifier is the best possible classifier for any supervised learning problem when you have enough training data (as proven by Stone, 1977). Thus, it shouldn't be too surprising that many different methods work very well when given massive training sets. We saw the same thing back in the 90's with the MNIST data set.

I agree with gcr that moving to new data sets makes sense, although performance on LFW with limited training data may continue to be interesting for some time.

-Erik Learned-Miller

Re:A Face Recognition Algorithm That Outperforms Humans? by df41 on Thursday April 24, 2014 @12:02AM (#7638447)
I am not in the area but Xiaoou is not nobody in the area. He used to work (or visit) at MS Research Asia and Chinese Academy of Sciences. He had best paper award from CVPR and served as chair of ICCV. That all being said, we all know how research goes... I am just curious why they put it on arxiv. Is this a common practice in the CV area or is it because someone else is working on a similar approach?
Re:A Face Recognition Algorithm That Outperforms Humans? by gcr on Thursday April 24, 2014 @10:23AM (#7640416)
Oh, that is a good point. I didn't see who the author was.
Re:A Face Recognition Algorithm That Outperforms Humans? by im3w1l on Wednesday April 23, 2014 @10:48PM (#7638258)
As comparison, the deepface paper also puts human performance at 97.5 %, and also isn't mentioned by Erik Miller. https://www.facebook.com/publications/546316888800776/

EDIT: The authors also seem to have presented a paper at ICCV 2013, so they aren't nobodies.

Re:A Face Recognition Algorithm That Outperforms Humans? by apu on Thursday April 24, 2014 @03:13AM (#7638870)
Generally, new results aren't added to LFW until after the conference takes place. CVPR will happen at the end of June this year.
Re:A Face Recognition Algorithm That Outperforms Humans? by tenfingers on Thursday April 24, 2014 @02:41AM (#7638808)
Since you work in the field, could you please recommend some free dataset for facial recognition?

I was trying to train an LBP classifier (mostly for fun), but I was disappointed to find out that I couldn't get my hands on some large dataset.

In particular, I was hoping to find not only a training dataset with the cropped frontal faces, but possibly also side faces, and then a large benchmark dataset that I could use as well to test the recognition accuracy.

It makes me mad to think that such datasets come basically for >free< for companies such as Facebook, but I couldn't get my hands on a free one that I could use to improve the OpenCV classifier...

Re:A Face Recognition Algorithm That Outperforms Humans? by gcr on Thursday April 24, 2014 @10:22AM (#7640409)
That is a great point. Some of this is due to privacy issues -- since my school's IRB board considers "Face recognition" to be "human experimentation," anyone who collects these datasets in a laboratory setting should go to the trouble to ensure that the subjects inside the datasets are treated fairly. Part of that means vetting who they give the dataset to. Sometimes there's bunch of red tap and silly rules about "When you show people dataset samples, you can only use these images... ..." Many of the high-quality pose-controlled sets require you to physically write to some student and request to enter into a license agreement with that institution, even though the dataset might be "free" to access.

There are other datasets from around the web that are much easier to get your hands on. LFW is honestly a great way to start, since it's freely available and since the "Results" page provides links to many state-of-the-art papers in the field (with PDF links for most of them, too!)

"YouTube Faces" is also another good one, containing videos: http://www.cs.tau.ac.il/~wolf/ytfaces/

Columbia has a few great datasets like "PubFig," which are distributed as lists of image URLs that you can download: http://www.cs.columbia.edu/CAVE/databases/pubfig/

See also: http://www.face-rec.org/databases/

If you do want something pose-controlled and high quality, you might just have to bite the bullet and write to the curators of "Multi-PIE" or "PIE" at CMU.

Re:A Face Recognition Algorithm That Outperforms Humans? by tenfingers on Saturday April 26, 2014 @10:13AM (#7651419)
Thank you for pointing out LFW, it's a very good dataset. YouTube faces also seems very interesting for benchmarking.
Re:A Face Recognition Algorithm That Outperforms Humans? by therobot24 on Thursday April 24, 2014 @07:03PM (#7643664)
Huzzah! Someone else recognizes GBU, i've been working with the FOCS dataset for some time, and wow is it difficult. My best result (using only the ocular region) is around 18% eer (note, with eer better is lower, my rank-1 id is easily +90%...but that's not a good measure of system performance IMO), Unfortunately it doesn't have the first name basis like LFW...
Re:A Face Recognition Algorithm That Outperforms Humans? by vlasev on Wednesday April 23, 2014 @10:47PM (#7638257)
For what it's worth, the graph (Figure 4) on page 10 features the label "Human, cropped 97.53%" and later again in the comparison section but not in the abstract. Still, the whole thing raises some red flags!
Re:A Face Recognition Algorithm That Outperforms Humans? by cybernoodles on Thursday April 24, 2014 @01:14AM (#7638615)
I just emailed Erik about adding it to the list.