OpenAI functions solves this problem for the GPT API. It's only a matter of time before the same functionality is available for open source models

So use better more standardized prompts. Specify the exact fields you demand of your unstructured data, refine your pipeline.

Just last week I went over 8k lines of data, doing a forst applicability analysis, meaning which lines to be considered for further analysis. The information I needed to do so was hidden in manually created comments, because of course it is, I have never ever seen pre defined classifications used consistently by people. And those pre defined classes never cover whatever need one has years later anyway.

Thing is, when I started I didn't even what to look for. I knew once I was done, so almost impossible to explain that to LLM before. Added benefit, I found a lot of other stuff in tue dataset that will be very useful in future. Had I used a LLM for that, I wouldn't know hald of what I know about that data I do now.

That's the risk I see with LLMs, already now my pet peeve are data scientist with no domain knowledge or understanding of the data they analyze, but at least they now the maths. If part ofbthat is outsourced to a blackbox AI that halluzonates half the time, I am afraid most of those analysises will be utterly useless, or worse, misleading in a very confident way...

Can you elaborate on the problem? I've successfully used LLMs to format unstructured data to json based on a predefined schema in a number of different scenarios yet somehow missed whatever issue you're describing in your comment.

Can you share your techniques or resources you used to get this working? We've got something that works maybe 90% of the time and occasionally get malformed json back from OpenAI.

Jsonformer - https://github.com/1rgs/jsonformer-claude, https://github.com/1rgs/jsonformer