Qwen2.5 Turbo extends context length to 1M tokens

Re:Qwen2.5 Turbo extends context length to 1M tokens by anon291 on Monday November 18, 2024 @02:27PM (#42175959)

> This means there are operations that are VERY HARD for the model like ranking/sorting. This requires the model to attend to everything to find the next biggest item, etc. It is very hard for the models currrently.

Ranking / sorting is O(n log n) no matter what. Given that a transformer runs in constant time before we 'force' it to output an answer, there must be an M such that beyond that length it cannot reliably sort a list. This MUST be the case and can only be solved by running the model some indeterminate number of times, but I don't believe we currently have any architecture to do that.

Note that humans have the same limitation. If you give humans a time limit, there is a maximum number of things they will be able to sort reliably in that time.

[ Parent | on HN ]

Re:Qwen2.5 Turbo extends context length to 1M tokens by christianqchung on Monday November 18, 2024 @05:17PM (#42177723)

Transformers absolutely do not run in constant time by any reasonable definition, no matter what your point is.

[ Parent | on HN ]

Re:Qwen2.5 Turbo extends context length to 1M tokens by anon291 on Monday November 18, 2024 @05:35PM (#42177845)

They absolutely do given a sequence size. All models have max context lengths. Thus bounded by a constant

[ Parent | on HN ]