I found them out myself when making our own implementation of the model. We test our outputs against upstream models. In decoding without history, our tests passed, but in decoding with history there was a mismatch between our implementation and the upstream implementation. Naturally, I assumed that our implementation was wrong (being the newer implementation, not sharing code with theirs), but while debugging this I found that our implementation is actually correct.

Then I was planning to report these issues. Someone else found the causal mask issue a week earlier, so there was no need to report it:

https://github.com/pytorch/pytorch/issues/103082

I reported the issue with rotary embeddings in a discussion of problems that people were running into trying to use KV caching:

https://huggingface.co/tiiuae/falcon-40b/discussions/48#648c...

More in general, I am not sure what the best place is to track these issues. Maybe a model's discussion forums?