Re:100x defect tolerance: How we solved the yield problem
by emkee on Thursday January 16, 2025 @09:25AM
(#42725507)
Having only taken one syntax class for fun in college, I find this pretty impressive. Generating syntax trees was never a trivial task for me (but I was just a CS major who needed a credit). Slightly related, but I have also never had ChatGPT successfully generate ASCII art, even with extensive conversation.
Re:100x defect tolerance: How we solved the yield problem
by stonemetal12 on Thursday January 16, 2025 @12:10PM
(#42727974)
If you ask it to draw a dinosaur it does an okay brontosaurs.
__
/ _)
_.----._/ /
/ /
__/ ( | ( |
/__.-'|_|--|_|
Asking for a Tyrannosaurus Rex gives you more or less the same brontosaurs: __
/ _)
_.----._/ /
/ /
__/ ( | ( |
/__.-'|_|--|_|
Re:100x defect tolerance: How we solved the yield problem
by kragen on Thursday January 16, 2025 @09:39AM
(#42725751)
Yeah, I think it would be very challenging for most people. It did considerably better with Graphviz than with ASCII art, but it still had trouble with the transition from a perfectly correct and highly nuanced verbal grammatical analysis to Graphviz. I think this is pretty convincing evidence against ChuckMcM's implicit position. It's weaker evidence against mschuster91's explicit position because parsing is something computers have been doing for a long time, so it doesn't imply any new capabilities.
I'm pretty sure there are part-of-speech tagging parsers using fairly shallow statistics that could also have produced an equivalently good sentence diagram. https://corenlp.run/ seems to produce a correct parse, though in a different format.
Re:100x defect tolerance: How we solved the yield problem
by CamperBob2 on Thursday January 16, 2025 @12:15PM
(#42728050)
(I know that it's generally rude to include LLM output in HN comments, but in this case I think it's essential supporting material to elevate the discussion of LLM capabilities above "yes it is", "no it isn't".)
You just have to be prepared to take a karma hit for it. The audience here does not consist largely of 'hackers', but seems to skew toward the sort of fearful, resentful reactionaries that hacker culture traditionally opposes.
I wouldn't have pegged ChuckMcM as being one of them, though. That's an unpleasant surprise.
Re:100x defect tolerance: How we solved the yield problem
by kragen on Thursday January 16, 2025 @04:39PM
(#42731253)
ASCII art Reed–Kellogg sentence diagrams are probably hard to find anywhere, and Graphviz can't really express Reed–Kellogg diagrams. But Reed and Kellogg published their somewhat ad-hoc diagram language in 01877, 78 years before what we now call "linguistics" was known in the West thanks to Chomsky's work in 01955. These are among the reasons I thought it might be a good idea to use the form of sentence diagrams used by linguists instead of the more compact Reed–Kellogg diagrams.
(I know that it's generally rude to include LLM output in HN comments, but in this case I think it's essential supporting material to elevate the discussion of LLM capabilities above "yes it is", "no it isn't".)
First, I just asked ChatGPT (GPT-4o mini, what they offer anonymous free users):
> Please diagram the sentence, "If my mother's road had been cultural, my uncle's sloth squad residue would have compiled a periodic meadow of night."
I just invented that sentence, picking several words with a random number generator, but of course I can't prove to you that I didn't copy it from some book you haven't seen. (For that, you'd have to invent a sentence yourself, or at least provide me with some words to include in it.) I hope you can at least agree that it's a highly unlikely sentence for anyone ever to have uttered before!
After talking to itself for a few paragraphs, it arrived at the following grammatical analysis:
This is exactly correct. From that it produced the following ASCII art "diagram", which looks like nonsense to me, but maybe it's in some kind of sentence-diagramming formalism I haven't seen: I told it the diagram looked like nonsense, and it produced this one, which looks worse: I asked GPT-4 (paid) the same question, and it gave me another exactly correct grammatical analysis in words: (I feel like this is definitely "a level that you'd assume to be competitive to a human", in mschuster91's phrase; most native English speakers can't do that.)But then it came up with the following incorrect diagram:
Telling it the diagram was nonsense didn't help.But my conclusion from this is not that GPT-4 can't analyze the grammatical structure of sentences it had never seen. It's that GPT-4 gets "confused" when it tries to draw ASCII art, like a person trying to tell you how to draw it over the telephone, but not quite as bad. I've seen similarly abominable results when it tried to draw circuit schematics and guitar tablatures as ASCII art. I tried asking it for a graphviz diagram, which was syntactically correct graphviz code but didn't express the sentence structure properly.
I tried simplifying the problem with the prompt, 'Please provide a syntactic tree showing the syntactic derivation of the sentence, "My uncle's sloth squad residue would have compiled a periodic meadow of night," using graphviz syntax and with individual words at the leaf nodes of the tree.' which yielded a slightly incorrect diagram. (I'm still sure nobody has ever uttered this sentence before.) On being told, "This diagram is not quite right.", GPT-4 produced the following diagram, which I believe is correct except for an extra "s" on "uncle" (you can see this in your browser at https://is.gd/Axufjx):
Claude produced a reasonably serviceable diagram on the first try with that prompt: On being told, I think incorrectly, "This diagram is not quite right.", it produced a worse diagram.So LLMs didn't perform nearly as well on this task as I thought they would, but they also performed much better than you thought they would.