News for hackers. Stuff that matters.
Re:Thorn in a HaizeStack test for evaluating long-context ad... by operator-name on Wednesday May 08, 2024 @12:26AM (#40294332)
If I've understood this correctly, the test is to measure the saftey finetune performance. These commercial models have been finetuned so that they are "safe", and safe models should not blindly quote what they are told.

Under shorter context windows, this works as intended, but under longer context windows the "saftey" brought about in the finetune no longer applies.

Re:Thorn in a HaizeStack test for evaluating long-context ad... by leonardtang on Wednesday May 08, 2024 @02:04AM (#40294817)
Bingo!