Then would know that visual question answering performs quite poorly. It makes for a nice demo, but it's trivial/quick/easy to find entire areas of questions where the models completely. The questions have to be worded very carefully. The models are quite finnicky and dumb. For example, dynamic memory nets aren't the state of the art - the state of the art for these datasets doesn't use anything nearly as complicated as memory, etc.
Do not underestimate what good ML model can learn given enough data. And in case of online shopping there usually is more than enough data, and big companies like Amazon have been using ML (context bandits) for quite some time.
1. https://arxiv.org/abs/1603.01417