Ask HN: What does your ML pipeline look like?

Re:Ask HN: What does your ML pipeline look like? by Liveanimalcams on Sunday April 21, 2019 @02:49PM (#19713787)

I'm not a professional but I built a pipeline for Makers Part List - It involves ingesting a video URL, converting the video into images, then storing the images in google storage. Once stored I trigger the model to classify the image. The images are then displayed to annotators who verify/relabel the images. Once I get enough new images the system creates a .csv and uploads it to googles autoML where it retrains my model.

My bottlenecks now are splitting the videos into images as its a very CPU intensive process. Implementing a queue here is my best choice I think.

[ Parent | on HN ]

Re:Ask HN: What does your ML pipeline look like? by mallochio on Sunday April 21, 2019 @03:07PM (#19713903)

Interesting. Could you maybe expand on the tools that you utilize inside the pipeline for ETL, model creation, annotation and testing?

[ Parent | on HN ]

Re:Ask HN: What does your ML pipeline look like? by Liveanimalcams on Sunday April 21, 2019 @04:31PM (#19714394)

I'm running the front and backend of the consumer site on Heroku. The meat of pipeline is hosted on a DigitalOcean High CPU Droplet. I use ffmpeg to extract images from the provided videos. I store everything in Google Cloud Storage and create references to each photo in Firestore. I use Firebase to power for the image verifying/labeling app I built. Its a simple app that presents the viewer with the image and the label that it was given. If its not correct they enter the correct label. I use a cloud function to move the images into an exportable format for autoML once a new image threshold has been hit. Testing is me using it and seeing if it is correctly identifying the objects.

[ Parent | on HN ]