A look at Apple’s new Transformer-powered predictive text model

Re:A look at Apple’s new Transformer-powered predictive text... by dchest on Sunday September 17, 2023 @07:28AM (#37543876)

There's no such thing as "base models have only the temperature setting". Models do not have sampling settings (temperature, repetition penalty, etc), the sampling code does, which obviously you can use on any model.

For example, here's a function from llama.cpp that applies repetition penalty: https://github.com/ggerganov/llama.cpp/blob/master/llama.cpp...

Here's the one from transformers: https://github.com/huggingface/transformers/blob/0a55d9f7376...

To summarize how they work: you keep some number of previously generated tokens, and once you get logits that you want to sample a new token from, you find the logits for existing tokens and multiply them by a penalty, thus lowering the probability of the corresponding tokens.

[ Parent | on HN ]

Re:A look at Apple’s new Transformer-powered predictive text... by cubefox on Sunday September 17, 2023 @07:42AM (#37543977)

I don't think such penalties were applied to GPT-2 or even GPT-3, yet they weren't repetitive like that.

[ Parent | on HN ]

Re:A look at Apple’s new Transformer-powered predictive text... by dchest on Sunday September 17, 2023 @07:52AM (#37544055)

Yes, they are applied. Here's OpenAI doc which describes how to set various sampling parameters for GPT-3:

https://platform.openai.com/docs/api-reference/completions/c...

See presence_penalty and frequency_penalty.

Sampling techniques is one of important arts of LLMs, you'll can find a lot of papers on them.

In general, smaller are more prone to repetition, but you can get caught in it even with larger models.

[ Parent | on HN ]

Re:A look at Apple’s new Transformer-powered predictive text... by dchest on Sunday September 17, 2023 @11:15AM (#37545909)

To clarify: I meant that in general they are commonly applied, but in this case they weren't as the author confirmed. The repetition, of course, doesn't happen all the time.

[ Parent | on HN ]

Re:A look at Apple’s new Transformer-powered predictive text... by jackcook on Sunday September 17, 2023 @08:06AM (#37544160)

Yes, you're right, I should have mentioned it in the post, but I used pure greedy sampling for the GPT-2 outputs since I couldn't do anything but that for the Apple model. So temperature was set to zero, and there was no repetition penalty.

[ Parent | on HN ]