"There's not enough good data to train it on properly"

Bingo: You're in _very_ data poor environment, compared to something like predicting a consumer's preferences in videos or identifying and segmenting a bicycle. The external data is also very qualitative and hard to encode into meaningful features.