Be careful with NER, while it is very good, it is not perfect.

Example, I tried to extract skills from a job posting. ChatGPT did well, but there were skills missing.

It is good to find some entities but then you need to extend the labeling manually.

For sure. If perfect accuracy is important, it's still good to do sampling and human review to figure out accuracy rate and decide if further checking is required. But it still beats the pants off accuracy of other methods for the amount of work required.

That said, with fine-tuning, `davinci-003` is _excellent_ at the types of entity extraction you're describing.

There is no such thing as davinci-003 per se. There is text-davinci-003 which you can't fine tune, and davinci which you can.

Sure, but SOTA on NER is around 90% for things like Names and Places. For skills, a Spacy model will get around 60-70%. And training a Spacy model takes a bit of fiddling.

Checkout Argilla for annotation. You can use vectors to speed up the annotations and you can also start with zero-shot feedback to improve the training as you go

https://www.argilla.io/

The cost may be high but still worth it depending on the alternative. I wonder what is happening with this in the evidence mining tools lawyers use for example.