Scaling Transformer to 1M tokens and beyond with RMT

Re:Scaling Transformer to 1M tokens and beyond with RMT by cced on Monday April 24, 2023 @08:33AM (#35686841)

Thank you very much for this comment. I just recently started dabbling with "AI/ML" by having Youtube videos transcribed and summarized. I was pretty amazed to see that I could simply 'plug' an .mp3 of audio and have openai/whisper transcribe the audio - locally - albeit at a non-trivial transcription time.

I was looking at getting those transcriptions summarized using a similarily wonderful tool but was kind of stumped when I started seeing 'maximum token length exceeded' errors.

I'm hoping to see some easy plug-n-play solutions to the summarization issue that run completely locally.

Fun times ahead.

[ Parent | on HN ]

Re:Scaling Transformer to 1M tokens and beyond with RMT by ca_tech on Monday April 24, 2023 @10:50AM (#35688355)

You can work around this by chunking the data using a sliding window. Take the first X amount of sentences that fit into the token length and summarize. Now slide your window of sentences so that you still overlap just a bit with your previous selection. Summarize that information. Continue until you have a bunch of summaries for your text. You can then pass that back into the model for a more concise summary of the summaries.

[ Parent | on HN ]

Re:Scaling Transformer to 1M tokens and beyond with RMT by cced on Monday April 24, 2023 @11:16AM (#35688682)

I’ll be exploring a combination of pre-processing such as stemming in order to also reduce the token length while preserving as much important information.

Thanks for the feedback.

[ Parent | on HN ]