Dennis Prager, a nationally syndicated talk show host, has requested such an app. He said he'd give an on-air interview to the person who makes it.

http://www.dennisprager.com/

Congratulations and well done. I'm just downloading it now. My first thoughts were if it was able to identify the 320+ species of native trees we have here in NZ. My second thought was to wonder if it can be trained by the users to learn new species? (I'm still waiting for the app to finish downloading so have not tried it yet). It would be great to expand your knowledge base by having user contributions, perhaps in a curated or weighted submission process.

Thanks! We currently have coverage of about 200 species from the northeast US. We're focusing on expanding the coverage to the rest of the US (~500 more species)...and then the world!

We definitely want to take advantage of user contributions in some way, but we're still trying to figure out the best thing to do. Also, the app contains several very high-quality images of each species taken in a particular way by a talented group called FindingSpecies, and to keep the visual integrity of the app, we would want very similar photos for all species that we add. This might be difficult to outsource to the crowd.

It's not something that can be readily outsourced to "the crowd" in general but it is something that can achieved using a hierarchy of interested/invested self checking power users.

It's a similar problem to that faced by, say, open source projects that aim to maintain a certain level of code quality and style matching in contributed source code.

Addendum: the image collections of the Australian National Botanic Gardens and the Australian National Herbarium might be of interest to you down the track: http://www.anbg.gov.au/anbg/photo-collection/index.html

I think this is awesome, I've been waiting for this technology for a long time. It's one of the first CV+Mobile apps I imagined having some day, and when you release the android version I'll finally get my wish! Thank you!

Question: Any thoughts to partnering with IQEngines to bring their adaptive image recognition engine and database together with yours?

We're computer vision people ourselves, so it's not an immediate priority =) Also, there was a lot of computer vision work we had to do specifically for this project to get the recognition to work. See elsewhere for my description of that.

I think it is an outstanding idea, CONGRATS for having done that!!.

It solves a real problem, maybe you can extend it to mushrooms with HUGE warnings(you don't want people eating something poisonous a computer recognized).

I have this little tree in my yard, I do not know for sure what it is, it seems it is a cherry tree but I'm dubious.

Mega Congrats! Any chance of developing a similar app (paid would be fine) for identifying wildflowers? I have thousands of pictures of wilderness areas across the Western US, and I get tired of saying "and here's an example of... a yellow flower I saw a lot of that day" :-)

We've thought about flowers, but since they are present on trees for a much shorter time than leaves, it would be less useful overall. Since we're somewhat constrained in our resources (most of the work has been done on a volunteer basis), we're focusing on expanding coverage first rather than integrating additional cues for recognition (which will be a very tricky computer vision problem that will likely require a lot of effort).

I have a family member that taught dendrology for 30 years.

He amazes me by being able to look at a leaf, bark or chunk of wood and tell what it is near instantly. If accurate, this is the sort of thing that will change the teaching process. I will tell him about it.

Any way I can get notified when the Android version comes?

You can follow @leafsnap on twitter. That's definitely the #1 request we've been getting, including from myself (I have a Nexus S) =)

Nice. You need to add achievements to this. :-)

Great work! This is an app I have been waiting for. I'm looking forward to you covering other plants, flowers and mushrooms.

Another app I am waiting for along the same lines is Shazam for bird song!

Terrific news! I read the 2009 article and have been wondering about it.

Is it open source?

It seems your site is having some trouble with the load.

Yes, had a manic hour, but things seem to be under control now. (I should have listened to patio11 the first 100 times he said to turn off KeepAlive.)

Also needed to cache a bit more aggressively.

Can you just tell us what the recognition tech behind it is?

Sure! There are 3 main pieces:

1. Segment the leaf from the background. Even though we require users to place the leaf on a white background, this is still a very tough problem because of lighting, shadows, viewpoint, blur, focus, etc. We've tried all published segmentation algorithms and nothing works. We wrote our own, using EM on the HSV colorspace as the basis for it (plus lots of heuristics for leaves). It works okay, but it definitely needs the most work.

2. Extract some features from the contour of the leaf. We use histograms of curvature over scale. The idea is that some leaves have the same rough shape, but differ at a fine scale (e.g., serrated vs. smooth leaves). Others are similar at a fine-scale but look quite different overall. To distinguish all of these cases, we estimate "curvature at a scale" at each point on the contour and build a histogram of curvature values. We then repeat this at coarser and coarser scales and concatenate all histograms together to get a large feature vector.

3. Recognition/matching: We use a simple nearest-neighbors classifier to match the feature vectors to our ground-truth database of labeled leaves. We return the top 20 ranked species or so.

A whole other approach, but I wonder if segmentation might be easier using a video of a leaf moving against a stationary background. The leaf is the part that is changing.

I've often wondered about identifying local trees; I've looked up site of trees, but often leaves seem very similar to several different trees.

Great to have it for other countries, and an edible plant guide.

Why are not using any color information in the feature vector? Does it result in poor performance improvement?

Color is very tricky to use, because you have to worry about white-balancing, illumination conditions, and of course leaves have a large variation in color over the course of their life.

This is really cool! I like it because I think it will actually encourage kids especially, and generally everyone, to take an interest in nature and the plants around them. Also, I'm sure it could be useful in identifying dangerous and safe plants. Keep up the good work!