Humans make mistakes - whether directly, or indirectly through the software that we craft. But when the mistake happens, we can reason about it, understand why it happened, and fix the problem.
Wouldn't it be embarrassing to not be in a position to reason about / explain a mistake to your customers caused by this "thing" we use that's been trained on all the random information available on the internet? Would that not reduce the value and standing of your business (which is ultimately about the people and expertise contained within) in the eyes of customers?
As an engineer, technologist, and CTO I absolutely love what I've been able to craft using generative AI under direct supervision.
But I cannot imagine a world where we give our carefully thought-through, easily changeable, algorithms and processes for a black box that cannot be reasoned about, no matter how good the output _sometimes_ is.
I have to think many people feel this way.
Usually, the list of recommendations are about ~8 services that we can replace. Our sales people know those services and prices by heart. A quick glance is enough for them. Our quotes are not legally binding. The contract that we sign is.
We found that it was highly accurate. GPT4 has been especially good at stuff like this after OpenAI changed it to write Python code for this kind of calculation.
just my 2 ct
For invoice parsing (various formats), are you just using GPT4V? When GPT4V initially came out, i benchmarked it against an out of the box invoice parser from Google Cloud (https://cloud.google.com/document-ai) on 16 documents and it was much better accuracy wise. For ex: i'd get results parsing 10,100 as 101100 (no comma).
Curious if you saw problems like this in your pipeline or if its gotten much better since?
Our sales people request invoices from a potential customer. On those invoices are our competitor's services and price. Invoices can come in PDF, png, jpeg, excel, csv, email formats. Content formatting can come in random forms. Pricing breakdowns are also non-standard across invoices. We have matching services and our own prices.
The goal is to find similar services where we charge less. In the past, our sales people would spend hours combing through those invoices. We wrote a prompt for GPT4, fed in our services and prices, and asked it to find services we could potentially replace as well as our profit margin. It took us a day to write this prompt. The results were outstanding and GPT4 gave accurate results. We even asked it to package it up in a PDF for us to send to the potential customers. On a Monday morning, we started on the prompt. By Tuesday morning, we got it working well enough that we were confident ship it to a few of our sales people to test.
This will save our company hundreds of thousands each year and we can get back to the potential customer much faster than before - increasing the likelihood of a sale.
If we had to program this like normal software, it'd probably take months to get it right with dedicated engineering resources to account for new invoice edge cases. Chances are, engineering would never even prioritize this feature for our sales people because there is simply no economical way to account for so many different invoice edge cases.
I believe we're just getting started. If we get GPT6 in two years and massive improvement in inference cost and context size, it's going to change everything we do. Heck, even GPT4 with 100x context size and 100x lower cost per inference would be transformative.
If this is a bubble, I'd like to live in it. I believe that many businesses have found use cases similar to the impact of ours. But they're just not broadcasting them to the internet in order to keep it a business advantage.