Hs position is that Color's value proposition kicks in when everyone's taking pictures all the time, blanketing every venue with visual documentation -- more photos, not better photos.
Flickr's value proposition is interestingness, not total coverage. Further, Flickr has three privacy layers, resulting in most night-on-the-town scenarios being Friends-only.
He made this point with his odds argument. Odds on Flickr might be one in a million, but Color wants users to take a million photos, making the odds one in one.
Probably the fact that Mrs. James has to search to find the picture on Flickr but the data is given to her with Color. Mrs. James only needs to be at the bar using Color and the images come straight to her. I see it as a subtle, but pretty substantial difference.
For the stated example of an adulterous husband on a business trip, you're right. The odds are low of Mrs. James finding it. But since pictures are grouped by physical location instead of residing in some stranger's flickr account, people are very likely to see people they know in cases when long-distance travel is not involved.
Is this a privacy issue? Strictly speaking, I don't think it is any less legal than flickr (but IANAL). But it definitely pushes the boundaries when it comes to thinking about your own personal privacy.
What privacy problem does color.com have that doesn't apply to flickr.com?