> So the longer you go in your generation, the higher the error rate. so at long contexts the answers become utter garbage.

Not totally wrong. They can self-correct, but it seems context rot will eventually set in.