Interestingly, this has been the approach of a dietary study being conducted by a group doing research on a circadian-focused pattern of eating. The study uses a mobile app as part of their distributed clinical trial design. Participants take pictures of their food with only the option of further journaling. Kind of cool.

Here's the website for the trial (& app)...

http://www.mycircadianclock.org/

Here's a video of the principal investigator (Satchin Panda) talking about his research...

https://www.youtube.com/watch?v=-R-eqJDQ2nU

Dissolved sugar is probably invisible or close. Can you see a difference between a glass of Coke and a glass of Diet Coke? One has 140 calories and the other has nominally zero.

That's not to say you couldn't approximate it and tell someone they ate a Twinkie today which is more fattening than the steamed broccoli they had yesterday. And you could probably help people manage their diet better with an app that could do that. But it's always going to be an approximation with a potential for large error in certain cases.

Recognizing food with ~90% accuracy isn't too hard. Getting calories from that is also easy. I'm currently building this into a food tracking app as a side project, with the hardest part being designing a UI that doesn't suck.