Re:Launch HN: UpTrain (YC W23) – Open-source performance mon...
by vvipgupta on Wednesday March 08, 2023 @11:37AM
(#35071169)
Additionally, refinement is a key focus of ours. Figuring out the best data points to retrain the model upon has twin benefits:
1) It provides automated issue resolution and saves data scientists' effort to debug and fix their models. 2) It allows us to reduce false positives in alerting: we send alerts only when we see a dip in model performance, or retraining can lead to improved model accuracy.
1. We use model-inferred embeddings. Say, for the instance segmentation task, we use deep learning networks to transform the input image into a dense embeddings representation, on top of which we run clustering and density estimation to find if the given embedding/image/feature combination is an outlier (or belongs to low-density region)
2. We allow users to define custom signals to identify edge-cases, specific to their use-case. A very simple example could be calculate brightness or Hue properties on the input image and see if that is an outlier compared to the training distribution.