News for hackers. Stuff that matters.
Re:Anthropic Claude 3.5 can create icalendar files, so I did... by sorokod on Sunday August 25, 2024 @05:02AM (#41345739)
Isn't asking an LLM "to write a Python validator" suffers from the 99.9% (or whatever the error rate for validators written by Claude) problem?
Re:Anthropic Claude 3.5 can create icalendar files, so I did... by bonzini on Sunday August 25, 2024 @05:25AM (#41345854)
The difference is that you're asking it to perform one intellectual task (write a program) instead of 100 menial tasks (parse a file). To the LLM the two are the same level of complexity, so performing less work means less possibility of error.

Also, the LLM is more likely to fail spectacularly by hallucinating APIs when writing a script, and more likely to fail subtly on parsing tasks.

Re:Anthropic Claude 3.5 can create icalendar files, so I did... by dbaupp on Sunday August 25, 2024 @05:57AM (#41346027)
In addition to what you say, it can also be easier for a (appropriately-skilled) human to verify a small program than to verify voluminous parsing output, plus, as you say, there's the semi-automated "verification" of a very-wrong program failing to execute.
Re:Anthropic Claude 3.5 can create icalendar files, so I did... by jessekv on Sunday August 25, 2024 @06:08AM (#41346091)
All tests have this problem. We still write them for the same reasons we do double-entry bookkeeping.