elandau25

Good question! We are focused on vision at the moment, but we are indeed looking at text in the future. Happy to connect and have a chat around that if you are open as we would be curious to hear more about new text use cases

Thank you! It was great working with you and your team as well :)

Thanks Kyle, appreciate it! Has been very nice collaborating with you!

Hello! I wrote the article so happy to answer this. It is partially feature engineering but partially not. It’s essentially using feature engineering to curate/correct a dataset, but a neural network as the actual end model without explicit input of these features(we call them quality metrics). I abbreviated a good amount of the process in the article so that it wouldn’t run forever, but essentially we allowed ChatGPT to select and write its own features and then used the strategies it came up with to apply these features to improve the dataset.

In terms of if it’s realistic in practice, the answer is yes. Some teams have a dearth of data, but many AI companies we work with have more data than they can use, and it’s more a question of how to sample, curate, and correct the data and labels they have to improve their models rather than collect new data. Great questions!

It is important to note that these micro-models are only supposed to be used in the annotation process. During annotation there is a separate process for QA where there will be some form of human supervision. Micro-models are NOT supposed to be used for production environments.

100% agree on the healthcare front, which actually perfectly underlies the point here. These models are overfit to one specific modality but often used for generic purposes. One reason why it is important to define micro-models is to point them out when they are deployed in a live production environment, which I agree is very dangerous. Many healthcare models are truly not ready for live diagnostic settings. On the other hand, these same models often do perform well on assisting the actual annotation of new data when applied to the right domain and with appropriate human supervision. This is a perfect encapsulation of the distinction we are trying to make.

Thanks! I might do a follow-up article on the topic and will think about how to incorporate this in!

That's a good idea. It will likely be trickier to apply a method like this for text. Decomposition of the problem is less obvious than with vision tasks.

Fair enough, but target data in this sense IS a full distribution of Batmen. This approach is towards the goal of creating a broad dataset and fitting a full Batman model. We are training on a narrower subset of our actual target data and fitting to that narrow subset, whether you want to call that overfitting or not I suppose depends on your perspective.

I agree intuitive definitions are often murky, but given we are already throwing in murky notions of intention that are implicit in the word "target", I think an at least colloquial usage of overfitting is appropriate.

Sometimes we try micro-models on broader domains than what we expect they will work for, and they work fine. Sometimes not. The point is that the target here is not well defined because we are just using them as annotation tools with some human supervision and not in a "typical" production environment.

If we are taking wikipedia as ground truth, the next line is:

An overfitted model is a statistical model that contains more parameters than can be justified by the data.

Another definition from https://www.ibm.com/cloud/learn/overfitting#:~:text=Overfitt...:

Overfitting is a concept in data science, which occurs when a statistical model fits exactly against its training data.

We can argue over the precise definition of overfitting, but when you fitting a high-capacity model exactly to the training data, that is a procedural question and I would argue falls under the overfitting umbrella.

I see what you are saying, but in that context then you lose what most people's intuitive definition of overfitting is. If I train a model on one image as my train set and then change one random pixel and run that model on this eval set then your argument would be that this is not overfitting because you are performing well on the eval set you created the model for.

My argument is that compared to models, as most people use them, micro-models are low bias and high variance, and thus overfit. That's why I set a distinction between a batman model and a batman micro-model.

Thanks! You have the application correct, but there are many ways by which we use this. An example is if you have trying to build models that require sequentially annotated images(like action recognition). Another is creating many micro-models that each only detect one type of object even though your general model will have to detect multiple objects.

In general, the theory of what you are saying is correct that this method annotates data that is correlated with the original set, but practically it is still quite useful. Having more ground truth to work with gives a lot more practical flexibility with things like sampling, testing your model, randomization, and training more robust versions of your model.

Hi everyone, I wrote the article. I do consider this overfitting because we are training on these frames way more time than would be normally advised for the size of the training set such that the error is essentially zero for these frames. The model performs well in "out-of-sample" here but only out of sample that is semantically close to the original training set. Besides, overfitting is defined procedurally, not by how well it performs. You could have an overfit model that just happens to perform well on some stuff it was not trained on, that doesn't change the fact that the model was overfit.

Great question. Superb AI also seems like a great tool. I’m not sure if they have video annotation though.

We are different from them in a bunch of ways, but the biggest one is that we are optimized for handling videos, sequential images, and radiology. We really like handling groups of semantically similar images.

Thank you! We think Hasty is a pretty neat tool, I am just not sure they have annotation for video datasets? Our platform is really optimized for video and sequential image. A GI rotoscope is right up our alley. If you have any more of those projects come up, give us a shout!

Really interesting idea with regards to the spectogram, especially with the image-like representation. Will take a look at some of those use cases.

You can do either! We offer a bunch of automation features directly through the Web App but people have also used the SDK to write their own algorithms. We have seen a lot of different annotation processes now so we can often direct people on the best flow to automate their labeling.

As far as I can tell Roboflow is more focused on being an end to end platform for AI. The customers we work with generally want to retain more control over their model building process, we just help them with automating as much of the annotation process as we can, often with the help of their own models.

Hi Eric from Cord here. Scale is a great company and they have done really well in AV especially. The issue with them is that they require you to send your data overseas to be annotated by a human workforce. They also probably have a bunch of in-house automated tools, but they don’t pass the savings onto their clients because they are paid on a per label basis. We want to pass the benefit of automation to anyone that needs labeled data.

Thank you!

I hear you, but I don't even think the labor intensiveness is a lost cause here. Labor intensiveness in mining insights from a dataset is worth way more than labor intensiveness in manual labelling.

I think we are lazily giving up our intellectual power to models hoping that they will just discover patterns by magic, where it is actually very worth to go through the data science process starting with labelling because we actually learn as humans. The thesis is that this will also make our DL models better in the long run. We would never have come up with cool algorithms if we just always outsourced this work to models.

Hi Sriku, with regards to your first point, not necessarily. I mentioned in another comment, but the model you are using the labels to build and the labelling process are related but not the same, they have different fundamental constraints and rely on different techniques. You can't have a human in the loop to help you out in a live model as an example.

You are right there are a bunch of difficult problems this technique isn't perfect for, but it actually can still help improve the efficiency of labelling a lot and when I do it I get the added bonus of understanding the dataset a lot better.

Feature extraction from a pre-trained model? Sure, again depends how you use it. We have used feature extraction + clustering for some of these labelling tasks successfully in the past.

Really just depends on the task. For this particular case I used a Faster-RCNN model with weights pretrained on the COCO dataset

Yea, so that's more of a comment on the accuracy of the auto generated labels, because this approach doesn't assume a different representative set of data than with human labelled data, just that less of the data is human labelled.

So it comes down to how good the auto generated labels are(from a human perspective), which is a fair point that I didn't address much in the article, but in general comes down to a good QA process(which is applied to both human labels and machine labels equally because humans also make mistakes in this stuff).

In the article the dataset was small enough and the labels simple enough that I could run very quick visual inspection over the results, but for more complicated tasks we have a more rigorous human review process for evaluating label accuracy(again to both human and algorithm produced labels). The auto generated labels may not be more efficient overall if they require a lot of correction after review, but for this case, and a lot of other ones, they just are empirically are.

I see it a bit differently. I see it as two separate(but correlated) tasks. There is labelling the data and building a robust model. There is a nuanced gap between the two. The labelling task and the model task live a different constraint space.

When you are labelling data, you have access to strategies and means that might not be available to your downstream model. In our experience this includes a human in the loop component, building non-robust ensemble models(we call these micro-models), and some "guess work" functions on the data. All of this together can make an "auto labeller" that does pretty well getting labels made, but really the sum of these strategies is very different from some well trained neural network that will be running on edge or whatever.

The point of a model is not to label the data, it's to generate some value in some out of sample task, quite different from strategies that you can run in a sandboxed environment with your training data.

Definitely easier said than done, but the process at least makes labelling interesting. Sometimes you run into roadblocks where you can't get past just having a human doing some element of the labelling, but once you do have a few algorithmic strategies that work reasonably well on a representative sample of your data, you can usually scale them pretty effectively to the rest of your data

Hi Imnimo, I wrote the article and definitely understand your concerns. The point is not the specific steps I took working in general for most datasets, but more the overall idea of using a more data science-y approach to labelling rather than just blindly throwing your data at a workforce.

A more varied dataset will require additional strategies. We have done this type of thing with various datasets and what normally works is a combination of some vertical models, heuristics specific to the dataset, classical computer vision techniques, and some human label seeding/correction.