News for hackers. Stuff that matters.
Re:Claude was trained on protein sequences at scale by octoberfranklin on Wednesday November 18, 2026 @02:21AM (#50542910)
Curious: how can there be a correct prediction for the next element of a "shuffled" protein? The shuffled proteins don't occur in real life, so what makes one answer correct and another incorrect?
Re:Claude was trained on protein sequences at scale by anonymous_sorry on Wednesday November 18, 2026 @02:32AM (#50542973)
It is a rather odd control. Is it just to confirm that Claude isn't reading the sequence off disk?
Re:Claude was trained on protein sequences at scale by gilleain on Wednesday November 18, 2026 @02:56AM (#50543148)
Perhaps. They explain why:

> Shuffling keeps the letters and destroys their order, so the difference in accuracy measures what the model knows about the sequence itself.

So by analogy, if you know a word in English starts with 'Q', you could reasonably predict the next letter would be 'u'. By shuffling, you might encounter the letter Q later on, and not successfully predict a 'u'.

Put another way, the shuffled sequences can only use frequency analysis for prediction, while the unshuffled ones have local and global order signals.

Re:Claude was trained on protein sequences at scale by octoberfranklin on Wednesday November 18, 2026 @03:11AM (#50543291)
Yes but what is the source of truth for the next element of a shuffled sequence?

For a natural sequence, the observed molecule in nature is the source of truth.

Re:Claude was trained on protein sequences at scale by gilleain on Wednesday November 18, 2026 @03:25AM (#50543422)
We are definitely going beyond the borders of understanding of sequences here, but I would guess the source of truth is just the expected frequency distribution of amino acids? So 'G' (glycine) is most frequent, and something like 'W' (tryptophan) is least frequent (dependent on organism?).

However I thought the point was that the _difference_ between the shuffled prediction and the unshuffled is the measure here. So shuffled is the baseline, and the gap above that is how good the prediction is.

Re:Claude was trained on protein sequences at scale by tstactplsignore on Wednesday November 18, 2026 @07:48AM (#50545677)
It is a negative control in the experiment. Shuffling is a better control than random because it accounts for the raw rates of amino acids in each protein. For example W is very rare, so Claude could beat a truly random generator just by guessing not W. But the signal seen on real sequences is much stronger than that, so it tells you that Claude is doing something more than just guessing 'not W' on real sequences.
Re:Claude was trained on protein sequences at scale by octoberfranklin on Wednesday November 18, 2026 @10:11AM (#50547265)
But the signal seen on real sequences is much stronger than that

But the shuffled sequences aren't real sequences!

Sorry if I'm being daft here, but I still don't understand what the source of truth is for the shuffled sequence tests.

Re:Claude was trained on protein sequences at scale by wavemode on Wednesday November 18, 2026 @10:43AM (#50547697)
[delayed]
Re:Claude was trained on protein sequences at scale by octoberfranklin on Wednesday November 18, 2026 @01:13PM (#50549870)
Oh! So a lower score on the "shuffled" sequence is more validating of the experimental hypothesis.

I wish the article had stated this clearly. It seems to imply the opposite, which is of course false.

Re:Claude was trained on protein sequences at scale by wavemode on Wednesday November 18, 2026 @10:33AM (#50547548)
[delayed]