It's most likely due to a lack of image training. For example it knows how streets and towns are connected but completely fails on giving left/right or compass directions. Same sorts of failures with geometry problems.