Cf Lies we believe about Names, Addresses and Dates.
Perhaps there’s something else in the post that you’re talking about but please do take the time to actually read what’s there, I’m only telling you what’s written in the text around the problem you think you’ve found.
It’s an interesting problem. And i have following thoughts to it?
Are all cases even decidable? Shouldn’t one false positive rank higher than a hundred rank negative. Shouldnt’t the AI just abstain from decisions to avoid false positives.
Like imagine sending a user an email about a purchase that is not his, or a user can see the order of another user?
No, and I think undecideable is quite frequent.
> -Shouldnt’t the AI just abstain from decisions to avoid false positives.
Yes, and I think regulation should be quite strong in this regard.
> - Is it smarter to seperate this in a lot of small sub problems and to define this as a rule set of small yes no decicions, why does this need to be solved agentically?
For larger contexts I worry about agentic confabulation, but humans unquestionably look at the gestalt when doing matching. I don't want idiot clankers making enforced decisions because they have misunderstood one field.
> - You should definitely clean up deterministically first.
If you can't do field validation on entry then I'm not sure deterministic rules work better, they may be worse, although easier to reason about.
I personally wouldn’t lean towards LLMs on identity resolution. I would like give it the dataset and then ask me questions about specs? I really struggle to find the LLM part reasonable here. I mean potentially some typos, or nicknames, but those could be potentially resolved deterministically also.
I would probably have reviewer running that suggests new rules that I could implement. To me this seems more like compiler work than LLM work.
>Yes, and I think regulation should be quite strong in this regard.
See also: The other Sean Byrne doesn't exist
https://news.ycombinator.com/item?id=49307592
As well as some of the harrowing comments in the thread.
The system doesn't return confidence values, but the models do.
>Without knowing the cultural context, the location, the document type, the document author, etc, you could be getting it wrong.
And? There's no perfect process, just ones that asymptotically approach correctness at exponentially increasing cost. They own the system and get to decide what works for them.
Rather than binary classification we should be locating document fragments in a high dimensional feature space, and with all those fragments coming to a confidence bounded guess.
Additionally, when someone submits documents that can't be recognised as valid, the response should not be INVALID, it should be UNDECIDED. In that sense, passing to Opus is great, but passing a tiny fragment of the problem is a bad way to do it.
They propose:
Valentino → Female Child
Yet this is obviously a boy name, else it would be Valentina. Is this model going to return confidence values? No, they want decisions. This is an inherent problem in AI with reductive logic for these tasks.
Likewise, dates. Is 17/08/22:
- 20170822 (Chinese write the year first, and in speech and chat they will say '17' not '2017'.)
- 20220817 (Everywhere But North America.)
Is 04/06/2009:
- 20090604 (Non-North American)
- 20090406 (American)
Without knowing the cultural context, the location, the document type, the document author, etc, you could be getting it wrong.