Mistral 7B Paper on ArXiv

Re:Mistral 7B Paper on ArXiv by mhartz on Wednesday October 11, 2023 @12:30PM (#37846595)

Can someone help me understand Figure 2? Why does the newest token appear at the beginning of the sequence rather than next to its neighboring token?

[ Parent | on HN ]

Re:Mistral 7B Paper on ArXiv by nivekkevin on Wednesday October 11, 2023 @01:10PM (#37847216)

it's a rolling buffer, so it just upsert index % 4 in this case

[ Parent | on HN ]

Re:Mistral 7B Paper on ArXiv by mhartz on Wednesday October 11, 2023 @01:14PM (#37847281)

Thanks, so does that mean position within the buffer is irrelevant?

[ Parent | on HN ]

Re:Mistral 7B Paper on ArXiv by nivekkevin on Wednesday October 11, 2023 @01:21PM (#37847382)

it does feel like so, the position eventually loses its meaning as more and more data gets crunched by the training process, eventually it's just a context of the past 4 tokens it feels like

[ Parent | on HN ]