Re:Models with Large Context Window
by PaulHoule on Thursday May 09, 2024 @08:48AM
(#40307638)
I use SBERT-type models for ranking, retrieval, classification, etc.
These work properly for a document that fits in the context window but not for a document which is larger. You can cut the document into smaller pieces but it just isn’t the same.
I would like to see longer context models like that and could accept some inference cost for it.
I think the summarization-type use cases will benefit from the context window but the computational complexity of a problem posed can grow explosively as a function of problem size: so x10 context window might not really mean it can do a x10 sized problem consistently.
These work properly for a document that fits in the context window but not for a document which is larger. You can cut the document into smaller pieces but it just isn’t the same.
I would like to see longer context models like that and could accept some inference cost for it.
I think the summarization-type use cases will benefit from the context window but the computational complexity of a problem posed can grow explosively as a function of problem size: so x10 context window might not really mean it can do a x10 sized problem consistently.