liquidki

I think this is the achilles heel of LLM-based AI: the attention mechanisms are far, far, inferior to a human, and I haven't seen much progress here. I regularly test models by feeding in a 20-30 minute transcript of a podcast and ask them to state the key points.

This is not a lot of text, maybe 5 pages. I then skim it myself in about 2-3 minutes and I write down what I would consider the key points. I compare the results and I find the AI usually (over 50% of the time) misses 1 or more points that I would consider key.

I encourage everyone to reproduce this test just to see how well current AI works for this use case.

For me, AI can't adequately do one of the first things that people claim it does really well (summarization). I'll keep testing, maybe someday it will be satisfactory in this, but I think this is a basic flaw in the attention mechanism that will not be solved by throwing more data and more GPUs at the problem.

To be sure, this is also taught in writing workshops, speaking workshops, comedy and improv as well, the rule of three:

https://en.wikipedia.org/wiki/Rule_of_three_(writing)

> One pernicious stat is that we are judged by graduation rate.

What were you judged on before this? How do you evaluate the quality of a teacher? My mom is an elementary school teacher and she talks about similar things that the schools are pushing hard to have the highest graduation rates.

That sounds like a dangerous stat to judge by, since as you and she are noting the teacher's jobs are now tied to how many kids pass (and that the number keeps going up), not how well they teach.

Kinda like giving home loans to people who have no ability to pay them back?