Let's say I give it as input the sentence:
I . . . . . . . . happily.
The second word to be predicted depends on the first word.
Sometimes people will attempt to parallelize this by using a faster model to guess a few tokens and then evaluating them in as a batch with the main model to determine whether the choices were good.
[0] Usually it outputs “logits”, which become a probability distribution when combined with a “temperature” parameter.
It isn't. There is no guarantee that successive tokens will be comprehensible.
> Usually it outputs “logits”, which become a probability distribution when combined with a “temperature” parameter.
The logits are the probability distribution (well technically, you would apply softmax). Temperature is a parameter for how you sample those logits in a non-greedy fashion.
I think temperature is better understood as a pre-softmax pass over logits. You'd divide logits by the temp, and then their softmax becomes more/less peaky.
probs = (logits / temp).softmax()
It's "obvious" only in hindsight.
Does the rejection sampling lead to a statistically correct sample from the joint probability distribution or is that just a (possibly rough) approximation?
Key quote from Sec 4: "In this section we describe an order-agnostic training procedure, DeepNADE (Uria et al., 2014), which will address both of the issues above. This procedure trains a single deep neural network that can assign a conditional distribution to any variable given any subset of the others. This network can then provide the conditionals in Equation 1 for any ordering of the input observations. Therefore, the network defines a factorial number of different models with shared parameters, one for each of the D! orderings of the inputs. At test time, given an inference task, the most convenient ordering of variables can be used."
If you disagree and wish it 'kept out', my suggestion would be to... not link it? Especially in other pages which have nothing to do with Conway?
https://x.com/gwernbranwen1/status/1742214526878265550
...you are in no position to make claims about how posterity will consider the bad things about other people, asshole.
I've done what I can to get that account blocked: I have reported him for impersonation to Twitter years ago, after his previous accounts, and I have asked my followers to report the account to get it banned, and I have asked again just now. If any HNers would like to take a moment to report the false troll https://x.com/gwernbranwen1 account for impersonation of my real Twitter account, https://x.com/gwern , I would be grateful - there must be some number of reports at which he'll finally be banned...
But Twitter takes forever to ban impersonation accounts, so, 'gwernbranwen1' keeps on tweeting that crap.
(Why, you might wonder? I don't know. I have no idea who or why whoever is behind that account does it. They have never explained it anywhere I've seen, nor do they seem to particularly hate me or are trying to ruin my reputation; their main fixation seems to be misogynist hatred of a few women like Julia Wise. I've met Julia Wise all of once, who is a nice woman who I liked, but I otherwise have no particular connection to her, and I have no idea why anyone would want to harass her online like that or why they would do so under my name. It's just the Internet, I guess: https://gwern.net/littlewood )
You can create libraries and sub libraries according to topic, and also create libraries for projects or reading lists. Can also create shared libraries, for your research team to share annotated papers.
Finally it can archive offline copies of web pages, which makes it useful for blog articles and other online resources that might vanish.
There's a little learning curve, but it's worth it if you find yourself juggling dozens of technical papers! Enjoy!
arXiv comes through a bit ugly atm but it's high on my to-do list. I'm leveraging the library that Firefox uses for reader mode, so most sites come through quite well. A lot of my work right now is expanding their metadata support and fixing parser issues.
The authors randomly permute (i.e., shuffle) input tokens in training and add two positional encodings to each token: one with the token's position and another with the position of the token to be predicted. Otherwise, the model is a standard autoregressive GPT. The consequences of this seemingly "simple" modification are significant:
* The authors can prompt the trained model with part of a sequence and then decode the missing tokens, all at once, in parallel, regardless of order -- i.e., the model can in-fill in parallel.
* The authors can compute conditional probability densities for every missing token in a sequence, again in parallel, i.e., densities for all missing tokens at once.
* The authors propose a rejection-sampling method for generating in-fill tokens, again in parallel. Their method seems to work well in practice.
I've added this to my reading list. Thank you for sharing it on HN.