thibaut-duguet

This is a good idea and there are actually 2 objectives when one wants to clean its dataset:

- you might want to optimize your time and correct as many errors as you can as fast as you can. Using several models will help you ion that case, adn that's actually what we've been focusing on so far.

- you might want to find the most ambiguous cases where you really need to improve your models as those edge cases are the ones causing the problems you have in production. Those 2 objectives are quite opposite. In the first case, you want to find the "easiest" errors, while in the other one, you want to focus on edge cases and you then probably need to look at errors with intermediate scores, where nothing is really sure..

There was indeed a manual review of the "potential errors" highlighted by our algorithm to determine is it was indeed an error in the data or if it was an error in the prediction. The 20% corresponds to the proportion of objects that was corrected with this manual review. So it's actually likely that some errors (that were not found by our algorithm) are still in our clean version of the dataset.

The process is actually a bit complicated but let me explain it to you. Once you are on a dataset, click on the label that you want and use the slider at the top right corner of the page to switch modes (we call it smart detection). You should then be able to access three tabs and the errors are listed in the False Positive and False Negative tabs (I've added a screenshot in the blogpost so that you can make sure to be at the right place). Let me know if you have any problem, thanks!

I've added screenshots of errors in the blogpost so that you have an idea of the errors we spotted. Let me know what you think of them.

Our platform is actually designed for enterprise companies, so we don't provide open access unfortunately.

To answer your first question, we had both bounding boxes added and removed, and depending on the dataset, the main type of error was different (I'd say it was overall more objectifs that were forgotten, especially small objects).

It would indeed be very interesting to see the impact of those improved datasets on driving, which is ultimately the task that is automated for cars. We've been working on many projects at Deepomatic not only related to autonomous cars, and we did see some concrete impact of cleaning the datasets beyond performance metrics.

I'm a Product Manager at Deepomatic and I have been leading the study in question here. To detect the errors, we trained a model (with a different neural network architecture than the 6 listed in the post), and we then have a matching algorithm that highlights all bounding boxes that were either annotated but not predicted (False Negative), or predicted but not annotated (False Positive). Those potential errors are also sorted based on an error score to get first the most obvious errors. Happy to answer any other question you may have!