See https://github.com/jolibrain/dd_performances you can reach 1fps on image classification and objects detection tasks on rpie3 with custom caffe and models.