News for hackers. Stuff that matters.
Re:Superhuman: What can AI do in 30 minutes? by netnomad8 on Sunday March 26, 2023 @05:52AM (#35312762)
AI can filter through thousands of Legal case files and court documents to pinpoint exactly what cases are relevant to a current case that a lawyer may be advising on! It took me exactly 30 minutes to go through 428 case files... :)
Re:Superhuman: What can AI do in 30 minutes? by bottled_poe on Sunday March 26, 2023 @06:01AM (#35312795)
How much better is this than a keyword search across the documents?
Re:Superhuman: What can AI do in 30 minutes? by WoodenChair on Sunday March 26, 2023 @06:03AM (#35312800)
And when it’s a probabilistic LLM instead of a keyword search, how do you know you didn’t miss anything?
Re:Superhuman: What can AI do in 30 minutes? by _puk on Sunday March 26, 2023 @06:11AM (#35312842)
As opposed to manually trawling through thousands of documents, based on a keyword that may, or may not, appear in the context of the case file?

It's a huge boon, but nothing is perfect. If it's that important that absolutely nothing is missed, maybe try multiple approaches concurrently. Take this as a value add, not a replacement.

Re:Superhuman: What can AI do in 30 minutes? by ben_w on Sunday March 26, 2023 @06:19AM (#35312880)
If you don't already grok all the legal jargon, the LLM is probably better than a keyword search.

I'm not a lawyer, so when I search British copyright law, I wonder why those forms of words don't result in the staff of search engines and social media sites being arrested and imprisoned:

https://www.gov.uk/government/publications/intellectual-prop...

Although, probably best to ask a lawyer for legal advice even if they end up using GPT-n themselves; they've probably got public liability insurance if they get something wrong.

Re:Superhuman: What can AI do in 30 minutes? by wickoff on Sunday March 26, 2023 @06:08AM (#35312830)
Humans miss things all the time. It's okay to make mistakes.
Re:Superhuman: What can AI do in 30 minutes? by WoodenChair on Sunday March 26, 2023 @08:39AM (#35313825)
It’s not replacing a human, it’s replacing a keyword search.
Re:Superhuman: What can AI do in 30 minutes? by dist-epoch on Sunday March 26, 2023 @06:58AM (#35313106)
I've seen people use "code words" exactly to avoid them being find by regulatory keyword searches.

AI can figure out from context, a keyword search will miss it. And no, you don't know the "code words".

Re:Superhuman: What can AI do in 30 minutes? by MagicMoonlight on Sunday March 26, 2023 @07:51AM (#35313485)
Keyword search doesn’t understand the context of what you’re asking for.
Re:Superhuman: What can AI do in 30 minutes? by Jabbles on Sunday March 26, 2023 @06:31AM (#35312952)
Can it? Or are you just imagining that it might be able to one day?
Re:Superhuman: What can AI do in 30 minutes? by SCdF on Sunday March 26, 2023 @07:45AM (#35313438)
Did it? How do you know it didn't confidently miss important documents?

My experience so far has just been asking chatgpt questions and then researching it myself to confirm what it says, so maybe I'm missing something. But, it has been confidently wrong on important details a large enough percentage (right now) to make it absolutely not a fire and forget tool.

The worst part is the confidence: it's like having a coworker that just straight up lies to your face randomly. Even if it's only 5% of the time you basically can't trust anything they say, and so you need to double check all of it.

This doesn't make it useless, but it means it lends itself to "hard to do but easy to verify" tasks. Which afaict your example is not: you can verify the documents it picked out are relevant, but not that the documents that it didn't, weren't.

Re:Superhuman: What can AI do in 30 minutes? by asah on Sunday March 26, 2023 @07:49AM (#35313469)
coworkers give me wrong answers a lot more than 5% of the time.
Re:Superhuman: What can AI do in 30 minutes? by hiq on Sunday March 26, 2023 @08:23AM (#35313704)
On the other hand I can usually come up with my own estimate of how trustworthy the answer is when a human gave it to me, e.g. thanks to:

* their reputation with respect to the question domain (if I ask a basic C++ question to a C++ expert I'll trust them)

* their own communicated confidence and how good they are at seeing their own shortcomings (if they say "but don't quote me on that, better ask this other person who knows more" it's fine)

5% of bad answers doesn't matter if 99% of these times I knew I should look further. ChatGPT and others are missing this confidence indicator, and they seem to answer just as confidently no matter what.

To be clear I don't see a fundamental reason why LLMs couldn't compute some measure of confidence (which will itself be wrong from time to time but with less impact) so I expect this to be solved eventually.

Re:Superhuman: What can AI do in 30 minutes? by og_kalu on Sunday March 26, 2023 @12:24PM (#35316243)
Base gpt-4 already did this.(confidence about something directly correlated with ability to solve problem/answer questions correctly) You can read the technical paper. But the hammer of alignment(RLHF) took it away.
Re:Superhuman: What can AI do in 30 minutes? by copperx on Sunday March 26, 2023 @02:04PM (#35317460)
How in the world did you feed GPT those 428 case files? There's nothing in the API to make it ingest that much info.
Re:Superhuman: What can AI do in 30 minutes? by fritzo on Sunday March 26, 2023 @03:43PM (#35318717)
I'm guessing the author is computing embeddings https://platform.openai.com/docs/api-reference/embeddings