I previously stumbled upon that problem in the context of object detection and eventually gave up with my specific problem. But it seems like this is a more general issue.