It’s essentially a question of did you take a look the code? There is comments in the codebase directly where the config values in question are being used. Eg.

> # repetition penalty from CTRL paper (https://arxiv.org/abs/1909.05858)

Now I’m assuming a base rate of knowledge for this to help, but in general I find diving into the code paths of open source models, usually is a good use of time.

there's no for dummies, this stuff is too new. there's a bunch of white papers to read; _attention is all you need_ is a good place to start, kinda.

if you troll HN for AI related stuff and just read read read, progressively googling the terms you don't understand, eventually the haze will lift and things will start to make some sense.

source: in the middle of doing that right now

> if you troll HN for AI related stuff and just read read read

Do you mean "troll" or "trawl"?

(I'm thinking of an old saying that the best way to get an answer isn't to ask a question, but to say the thing is impossible. Can't remember it well enough to google for the exact form, however).

IIRC, both forms work.

Trolling is done by hand with a fishing pole, sometimes on a dock.

Trawling is from the back of a boat, generally with a net.

Interesting. Until now I only knew the verb "troll" as in "being mean on the internet".

If I'm not hallucinating like GPT, I think the etymology of troll as in internet actually stems from trolling as in fishing.

I have a hazy recollection of folks in the early days of 4chan describing 'trolling for idiots' - as in, feigning ignorance to get a rise out of folks. Like, you post a dumb, obviously stupid but infuriating comment and watch people get all worked up about it. You know, for the lulz.

I feel like since then it's changed to just actually being a dumb asshole and enjoying when people are mad at you. Or maybe I've just gotten smarter. In any event at the time it certainly didn't seem like the kind of thing racist lunatics would go out of their way to do. Perhaps more the fool I, as I was a teenager at the time and as such that opinion cannot really be trusted.

I think the best way is to get a basic understanding of torch and then just reading what the code does. The model internals are hard (though for the transformer there are good step by steps online) but for this case it is basically playing with the output of the actual neural net inside the model. The net gives you a probability distribution to be next over all tokens (think words). Now it has proven to be useful not to pick the token with the highest probability but to employ different strategies. Top-k in this case means probabilistically sampling from the top k tokens instead of piking the best.

"Transformers for Natural Language Processing", Denis Rothman (Packt)

To understand how pytorch works (bottom-up with practice video lectures): Neural Networks: Zero to Hero Andrej Karpathy https://youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9Gv...